Lisp as the Maxwell’s equations of software

Lisp as the Maxwell’s equations of software

Lisp as the Maxwell’s equations of software – DDI

A take on Lisp as the computational version of fundamental equations in physics. The claim is that learning Lisp is a foundational skill, and this page goes on to develop a “TiddlyLisp” interpreter in Python. As you’d expect this isn’t an espcially practical Lisp: but it’s remarkably functional, and I suspect will demystify Lisp for programmers familiar with interpreters for other languages.

See also a conversation with Alan Kay where he uses the “Maxwell’s equations”:

That was the big revelation to me when I was in graduate school – when I finally understood that the half page of code on the bottom of page 13 of the Lisp 1.5 manual was Lisp in itself. These were “Maxwell’s Equations of Software!” This is the whole world of programming in a few lines that I can put my hand over.

And in the second half of this article, the Lisp interpreter in Python is translated into a Lisp interpreter in Lisp, which is a very concrete way of showing how metacircularity can work in McCarthy’s original style.

Guard methods on CLOS generic functions

Guard methods on CLOS generic functions

There are times when one wants to be able to guard a method’s execution. A typical case is for callbacks, where we only want the callback to run under certain circumstances – but it’s easier to write the callbacks themselves as though they’ll always be called.

Object-oriented programs typically use a pattern for this: they split the function into two methods, one for the guard and one for the action being guarded. A sub-class can then override the guard independently of the action, and some sub-classes may override both guard and action.

This splitting seems a little awkward, though, and there are times when I’d prefer to have everything (guard and action) defined as part of the one method. Fortunately there’s a Lisp-ier solution involving defining a new method combination to get exactly this behaviour.

Standard method combination

CLOS, unlike most languages, allows a programmer to control how methods are combined in terms of overriding. The “standard” combination allows for :before, :after, and :around methods as well as undecorated “primary” methods.

When a generic function is called, the list of applicable primary methods is determined based on the types of arguments. most specific method first1.

The same process is performed for all applicable :before methods, and then again for applicable :after methods, and then again for :around methods. The :after and :around are always ordered most specific first, while the :before methods are always ordered least-specific first.

Once these lists have been constructed, the “effective” method that results is called. If there are :around methods, they are called in order. An :around method may, as part of its body, call call-next-method to invoke the next-most-specific :around method – or may not.

If a call to call-next-method has no more :around methods to call – or of there were no :around methods defined – all the :before methods are run and their return values discarded. Then the primary methods are run in the same manner as :around methods, with any calls to call-next-method calling the next primary method. After the primary methods have returned, all the :after methods are run and their return values discarded. The result of the method call is the result returned from the primary methods. The process is roughly like:

     (if arounds
         (call-method (car arounds) (cdr arounds)))

     (prog1
         (progn
           (call-methods befores)
           (call-method (car primaries) (cdr primaries)))
       (call-methods afters))

In this code, call-method calls its first method argument, and any call to call-next-method calls the next method in the list. call-methods calls all the methods in a list of methods, discarding their return values.

Contrast that with Java or Python, where methods on more-specific classes override those on less-specific, and have the option to call up to the superclass method. Essentially this makes all methods similar to :around, and there’s no real equivalent of :before and :after.

Other method combinations

The above is referred to as standard method combination, implying the existence of non-standard combination. CLOS lets the programmer define new combinations, and indeed defines a few itself. For our purposes the most important alternative method combination is and, which runs all primary methods within an and form treating all methods as predicates. There are only primary methods allowed.

Guards as method combination

For our use case, we want to be able to return values from primary methods, and allow :around, :before and :after methods. However, we also want to have some methods act as predicates that guard the execution of the effective method thus formed. We want to be able to add guard methods that are always run first, regardless of their specificity, and then run the effective method only if all the guards are satisfied. The net result is that all parts of the generic function are provided as methods on it, but some can now be boolean guards that act as gatekeepers on the rest of the methods. Naturally we want the guards to be selected for specificity alongside the other methods, letting the CLOS machinery pick all the functionality that’s appropriate to a particular method call.

Why this isn’t just :around

It might sound like we can get this behaviour using :around methods that perform guarding. But we can’t – quite.

Suppose we define a primary method:

     (defmethod example ((v integer))
       (* v 2))

We can write a guard quite happily as an :around method:

     (defmethod example :around ((v number))
       (when (> v 10)
         (call-next-method)))

This method will only allow the method to proceed when the condition holds, otherwise it returns nil.

     (list (example 26) (example 2))

(52 NIL)

So far so good.

However, the problem is that CLOS orders the :around methods most specific first. Suppose we have another :around method specialised against a more specific type:

     (defmethod example :around ((v integer))
       (if (= v 5)
           (+ v 1)
           (call-next-method)))

When this method is called with an integer this method gets run before the previous guard:

     (example 5)

6

and we get a non-nil result, despite the guard method indicating that we shouldn’t. If we provide an argument that doesn’t trigger the first :around method, then we can get caught by the guard:

     (example 6)

NIL

This is of course perfectly sensible behaviour in many cases. However, it does mean that the “guards” we’re supplying are executed as part of the effective method rather than before it, and therefore can’t guarantee that the method is properly guarded by all the guards, regardless of their specialisation. Another way of looking at this is that a later, more specialised, “guard” can override one set by an earlier, less specialised, method, which again may not be what’s desired.

A guarded method combination

Fortunately we can get the behaviour we want by defining a new method combination, guarded. A guarded generic function accepts five method qualifiers:

  • undecorated primary methods;
  • :before and :after methods that run before and after the primary methods;
  • :around methods that run around the :before-primary-:after combination; and
  • :if methods that act as guards, running before any :around methods to determine whether any of the “functional” methods are run or not

We first need a helper function2 to construct the code to run the chain of :before and :after methods while discarding their return values.

     (defun call-methods (methods)
       "Return `call-method' forms for all METHODS."
       (mapcar #'(lambda (m)
                   `(call-method ,m))
               methods))

We can then use the macro define-method-combination to define our new method combination.

     (define-method-combination guarded (&optional (order :most-specific-first))
       ((arounds (:around))
        (ifs (:if))
        (befores (:before))
        (primaries () :order order :required t)
        (afters (:after)))

       (let* ((before-form (call-methods befores))
              (after-form (call-methods afters))
              (primary-form `(call-method ,(car primaries) ,(cdr primaries)))
              (core-form (if (or befores afters (cdr primaries))
                             `(prog1
                                  (progn
                                    ,@before-form
                                    ,primary-form)
                                ,@after-form)

                             `(call-method ,(car primaries))))
              (around-form (if arounds
                               `(call-method ,(car arounds)
                                             (,@(cdr arounds)
                                              (make-method ,core-form)))

                               core-form)))

         (if ifs
             `(if (and ,@(call-methods ifs))
                  ,around-form)

             around-form)))

The macro is described in detail in the hyperspec, but its behaviour is quite simple. The list of forms (arounds and so on) define variables that extract the methods that have the given decorations – so arounds gets a list of :around methods, primaries gets the undecorated (primary) methods, and so on. In particular, ifs gets any methods decorated with :if: these are the guards.

The body of the macro constructs the code needed to build the methods’ behaviours. The let* defines the code for the different parts. core-form is slightly optimised in the case when there is only one primary method; otherwise it runs the :before methods and then the primary method, captures the result of the latter, then runs the :after methods, and then returns its result. (This is the first time I’ve ever used prog1 for real: now I know why it exists.) If there are :around methods, around-form wraps up a list consisting of the :around methods and a method constructed from core-form, letting it be run as the result of the final call-next-method call.

The body of the let* wraps-up around-form within an if whose condition is the conjunction of all the :if methods. Only if all these methods return true (well, not nil in the usual Lisp style) will the code of around-form be executed. Again the code is optimised for the case where there are no guards, in which case we just get around-form.

Notice that define-method-combination returns code, like all macros: it doesn’t execute the methods itself. This is a hint as to what happens off-stage: CLOS uses the method combination at compile time to construct effective methods which can then be cached to minimise the performance hit from all the flexibility provided by method combination.

Now we can re-do our example from above:

     ;; a generic function defined to use our new method combination
     (defgeneric guarded-example (v)
       (:method-combination guarded))

     ;; the functionality, split into two methods
     (defmethod guarded-example ((v integer))
       (* v 2))

     (defmethod guarded-example :around ((v integer))
       (if (= v 5)
           (+ v 1)
           (call-next-method)))

     ;; this guard used to be :around and is now :if
     (defmethod guarded-example :if ((v number))
       (> v 10))

     (list (guarded-example 26) (guarded-example 2) (guarded-example 5))

(52 NIL NIL)

The guard now stops execution of the effective method if its condition isn’t met – and if it is met, passes control through to the complete method stack. This happens regardless of where the guard is specialised in terms of the class hierarchy: the guards run before any “functional” code. (That :before and :after methods work too, and multiple guards, and that the combination works when applied to class hierarchies, are left as exercises to the reader.)

Critique

You may object to this solution on the grounds that it introduces a weird asymmetry into methods: some as functional and some as guards, with different return types. Maybe you prefer to keep guards in separate methods using the usual object-oriented pattern. That’s entirely reasonable. But I think there are sufficient cases where this kind of guarding makes sense to have it as a pattern, especially as it has no effect unless explicitly selected for a generic function.

I have to say I’m amazed how little code is needed: around 30 lines, including the helper function. It shows off the power of CLOS, and how it’s possible to change even the basic underlying structures of the object system with relative ease. But it also shows how Lisp opens-up the space of programming styles, things that benefit from being policies that can be changed, rather than hard-coding one particular choice.

Footnotes:

1

This is also programmable when required, for example to run methods least-specific-first.

2

I got the idea for this function from method-combination-utilities, and included it literally to avoid creating another dependency.

The Himmler Brothers: A German Family History

Katrin Himmler (2005)

The family history of Heinrich Himmler and his two brothers Gebhard and Ernst, as told by his great-niece.

There’s a lot of fascinating backstory in this book, and it explains a lot. It portrays Heinrich Himmler as very much the product of a middle-class, socially ambitious family who pushed him relentlessly to succeed and only seemed really to embrace him after he achiveed visible success as a Reichstag deputy and then later as Reichsführer-SS. The author, Katrin, has access to lots of family papers and photographs that have never been explored before, as well as being able to talk informally to Heinrich’s brothers’ children about their experiences. This makes this a profoundly personal exploration of Heinrich’s rise and fall.

That’s also its weakness, of course, of which Katrin herself is well aware: it was cleaerly not an easy book to write, and one led her to realise how much families re-write their own histories – understandbaly so in the light of what happened. It also means the the book as a whole concentrates on the domestic side of Heinrich’s life, which can be jarring: the period from him being a secretary to leading the SS is covered in less than a page. Overall, though, there’s a lot of keen insight provided into the making of a monster.

4/5. Finished Thursday 22 August, 2024.

(Originally published on Goodreads.)

Prophet Song

Paul Lynch (2023)

A timely and troubling consideration of social breakdown.

The most notable thing about this novel is that it happens in Ireland: not “far away”, but in a European country that is nevertheless seen as collapsing through (it is hinted) a shift to the right taken by fully democrtic means, and a subsequent descent into authoritarianism. That’s a perfectly believable scenario for many countries.

Lynch writes in a stream-of-consciousness style that manages to be comprehensible while still retaining the slightly bewildered feeling of someone caught-up in event they don’t fully understand and can’t quite keep up with. There’s enough Irish vernacular and geographical detail to make the story hard-hitting for anyone who knows Dublin.

Some people desperately cling to processes that no longer have any meaning; there are huge numbers of forms to fill out despite them serving no purpose; there’s lots of bureaucracy used to drum people into compliance; and there’s lots of venality hiding behind the continuation of these processes to convert them into vehicles for personal gain. Rules tighten as order breaks down, and the same cast of chancers and bandits and people-smugglers emerge in Ireland as in other countries that have gone through this kind of trauma. And that I think is the main message of the book. The themes and characters emerging from social breakdown are universal, and a European country would go the same way as anywhere else.

5/5. Finished Tuesday 20 August, 2024.

(Originally published on Goodreads.)

Fancy Bear Goes Phishing: The Dark History of the Information Age, in Five Extraordinary Hacks

Scott J. Shapiro (2023)

A lawyer’s take on hacking. That’s both a compliment and a limitation.

The book presents itself as being a history of hacking. It kind of is, and covers some famous incidents in varying degrees of detail. It spends a lot of time making the underlying technology understandable to a non-technical reader, which is a major contribution. One could argue that it’s mis-named, in that it spends more time on the hacking of Paris Hilton’s cellphone and the evolution of the Mirai botnet than on Fancy Bear’s attack on the US Democrats., but it does place these attacks into a wider and useful context.

But the wider context is als0 a problem. Shapiro is a legal philosopher – and it shows. Firstly he annoyingly introduces terminology to distinguish between computer code (“downcode”) and social norms and the law (“upcode”), with “metacode” sitting in between. Is this distinction useful? – I don’t think so, since the whole point about “upcode” is that it’s not code but a set of political choices and norms. Secondly, he tries to draw a distinction between criminal use of computers and espionage, arguing that the former is illegal but that the latter is not – and from this tries to argue that US responses to state-directed hacking are hypocritical. Well, yes: is that a point worth making, and does it have any significance beyond the purely philosophical?

Thirdly, the book also has a somewhat rosy view of US behaviour which is (in my opinion) contradicted by the facts and by the arguments of other books – for example This Is How They Tell Me the World Ends: The Cyberweapons Arms Race, which deals with the origins of cyberweapons.

There are also some annoying wild claims. Turing’s results absolutely don’t say that finding bugs in code is undecidible; this doesn’t stop us from radically improving our technological approaches to malware; we don’t think that problems are decidable because they’re all that we see; and a belief in the fundamental comprehensibility of the universe is not absurd. These weaken the presentation and detract from its undounbted strong points.

3/5. Finished Monday 19 August, 2024.

(Originally published on Goodreads.)