Practical example of Lisp's flexibility? [closed]

Someone is trying to sell Lisp to me, as a super powerful language that can do everything ever, and then some.

Is there a practical code example of Lisp's power?
(Preferably alongside equivalent logic coded in a regular language.)

I like macros.

Here's code to stuff away attributes for people from LDAP. I just happened to have that code lying around and fiigured it'd be useful for others.

Some people are confused over a supposed runtime penalty of macros, so I've added an attempt at clarifying things at the end.

In The Beginning, There Was Duplication

(defun ldap-users ()
  (let ((people (make-hash-table :test 'equal)))
    (ldap:dosearch (ent (ldap:search *ldap* "(&(telephonenumber=*) (cn=*))"))
                   (let ((mail  (car (ldap:attr-value ent 'mail)))
                         (uid   (car (ldap:attr-value ent 'uid)))
                         (name  (car (ldap:attr-value ent 'cn)))
                         (phonenumber (car (ldap:attr-value ent 'telephonenumber))))
                      (setf (gethash uid people)
                            (list mail name phonenumber))))
    people))

You can think of a "let binding" as a local variable, that disappears outside the LET form. Notice the form of the bindings -- they are very similar, differing only in the attribute of the LDAP entity and the name ("local variable") to bind the value to. Useful, but a bit verbose and contains duplication.

On the Quest for Beauty

Now, wouldn't it be nice if we didn't have to have all that duplication? A common idiom is is WITH-... macros, that binds values based on an expression that you can grab the values from. Let's introduce our own macro that works like that, WITH-LDAP-ATTRS, and replace it in our original code.

(defun ldap-users ()
  (let ((people (make-hash-table :test 'equal))) ; equal so strings compare equal!
    (ldap:dosearch (ent (ldap:search *ldap* "(&(telephonenumber=*) (cn=*))"))
                   (with-ldap-attrs (mail uid name phonenumber) ent
                       (setf (gethash uid people)
                             (list mail name phonenumber))))
    people))

Did you see how a bunch of lines suddenly disappeared, and was replaced with just one single line? How to do this? Using macros, of course -- code that writes code! Macros in Lisp is a totally different animal than the ones you can find in C/C++ through the use of the pre-processor: here, you can run real Lisp code (not the #define fluff in cpp) that generates Lisp code, before the other code is compiled. Macros can use any real Lisp code, i.e., ordinary functions. Essentially no limits.

Getting Rid of Ugly

So, let's see how this was done. To replace one attribute, we define a function.

(defun ldap-attr (entity attr)
  `(,attr (car (ldap:attr-value ,entity ',attr))))

The backquote syntax looks a bit hairy, but what it does is easy. When you call LDAP-ATTRS, it'll spit out a list that contains the value of attr (that's the comma), followed by car ("first element in the list" (cons pair, actually), and there is in fact a function called first you can use, too), which receives the first value in the list returned by ldap:attr-value. Because this isn't code we want to run when we compile the code (getting the attribute values is what we want to do when we run the program), we don't add a comma before the call.

Anyway. Moving along, to the rest of the macro.

(defmacro with-ldap-attrs (attrs ent &rest body)
  `(let ,(loop for attr in attrs
         collecting `,(ldap-attr ent attr))
     ,@body)) 

The ,@-syntax is to put the contents of a list somewhere, instead of the actual list.

Result

You can easily verify that this will give you the right thing. Macros are often written this way: you start off with code you want to make simpler (the output), what you want to write instead (the input), and then you start molding the macro until your input gives the correct output. The function macroexpand-1 will tell you if your macro is correct:

(macroexpand-1 '(with-ldap-attrs (mail phonenumber) ent
                  (format t "~a with ~a" mail phonenumber)))

evaluates to

(let ((mail (car (trivial-ldap:attr-value ent 'mail)))
      (phonenumber (car (trivial-ldap:attr-value ent 'phonenumber))))
  (format t "~a with ~a" mail phonenumber))

If you compare the LET-bindings of the expanded macro with the code in the beginning, you'll find that it is in the same form!

Compile-time vs Runtime: Macros vs Functions

A macro is code that is run at compile-time, with the added twist that they can call any ordinary function or macro as they please! It's not much more than a fancy filter, taking some arguments, applying some transformations and then feeding the compiler the resulting s-exps.

Basically, it lets you write your code in verbs that can be found in the problem domain, instead of low-level primitives from the language! As a silly example, consider the following (if when wasn't already a built-in)::

(defmacro my-when (test &rest body)
  `(if ,test 
     (progn ,@body)))

if is a built-in primitive that will only let you execute one form in the branches, and if you want to have more than one, well, you need to use progn::

;; one form
(if (numberp 1)
  (print "yay, a number"))

;; two forms
(if (numberp 1)
  (progn
    (assert-world-is-sane t)
    (print "phew!"))))

With our new friend, my-when, we could both a) use the more appropriate verb if we don't have a false branch, and b) add an implicit sequencing operator, i.e. progn::

(my-when (numberp 1)
  (assert-world-is-sane t)
  (print "phew!"))

The compiled code will never contain my-when, though, because in the first pass, all macros are expanded so there is no runtime penalty involved!

Lisp> (macroexpand-1 '(my-when (numberp 1)
                        (print "yay!")))

(if (numberp 1)
  (progn (print "yay!")))

Note that macroexpand-1 only does one level of expansions; it's possible (most likely, in fact!) that the expansion continues further down. However, eventually you'll hit the compiler-specific implementation details which are often not very interesting. But continuing expanding the result will eventually either get you more details, or just your input s-exp back.

Hope that clarifies things. Macros is a powerful tool, and one of the features in Lisp I like.

The best example I can think of that is widely available is the book by Paul Graham, On Lisp. The full PDF can be downloaded from the link I just gave. You could also try Practical Common Lisp (also fully available on the web).

I have a lot of unpractical examples. I once wrote a program in about 40 lines of lisp which could parse itself, treat its source as a lisp list, do a tree traversal of the list and build an expression that evaluated to WALDO if the waldo identifier existed in the source or evaluate to nil if waldo was not present. The returned expression was constructed by adding calls to car/cdr to the original source that was parsed. I have no idea how to do this in other languages in 40 lines of code. Perhaps perl can do it in even fewer lines.

You may find this article helpful: http://www.defmacro.org/ramblings/lisp.html

That said, it's very, very hard to give short, practical examples of Lisp's power because it really shines only in non-trivial code. When your project grows to a certain size, you will appreciate Lisp's abstraction facilities and be glad that you've been using them. Reasonably short code samples, on the other hand, will never give you a satisfying demonstration of what makes Lisp great because other languages' predefined abbreviations will look more attractive in small examples than Lisp's flexibility in managing domain-specific abstractions.

There are plenty of killer features in Lisp, but macros is one I love particularily, because there's not really a barrier anymore between what the language defines and what I define. For example, Common Lisp doesn't have a while construct. I once implemented it in my head, while walking. It's straightforward and clean:

(defmacro while (condition &body body)
  `(if ,condition
       (progn
         ,@body
         (do nil ((not ,condition))
           ,@body))))

Et voilà! You just extended the Common Lisp language with a new fundamental construct. You can now do:

(let ((foo 5))
  (while (not (zerop (decf foo)))
    (format t "still not zero: ~a~%" foo)))

Which would print:

still not zero: 4
still not zero: 3
still not zero: 2
still not zero: 1

Doing that in any non-Lisp language is left as an exercise for the reader...