Metacircular Semantics for Common Lisp Special Forms

Metacircular Semantics for Common Lisp Special Forms

Henry G. Baker. Metacircular Semantics for Common Lisp Special Forms. ACM SIGPLAN Lisp Pointers V, pp.11–20. 1992.

A response to the criticisms of McCarthy’s semantics for Common Lisp that it didn’t specify the behaviours of special forms, and of the standardisation process of adopting prose definitions that are too “lawyerly” for engineering.

The approach is to define the special forms in terms of other constructions, for example if in terms of nested lambda abstractions to prevent execution of the unwanted branch. This is both useful for understanding and a way of minimising the number of under-defined special forms.

It suggests treating catch / throw as basic, both because other control-transfer forms can be expressed sing them and because it emphasises the interactions that preclude Common Lisp having continuations like Scheme’s. However there’s also an argument pursued that some other structures (like values) provide extra information that can be useful for compilers looking to optimise. It’s a deep exploration of the underpinnings of the language from both theoretical and practical perspectives.

Casting SPELs in Lisp

Casting SPELs in Lisp

https://www.lisperati.com/casting.html

A tongue-in-cheek – but still excellent – comicbook-style introduction to Lisp programming. (There’s also a version specifically for Emacs Lisp.) It’s structured around building an adventure-style game, which as well as being a classic also offers lots of opportunities for exploring different data structures and algorithms: one can easily imagine expanding it to include (for example) an AI second player or autonomous non-player characters and gradually building a really complicated application from a standing start.

My mental model of setf was wrong

My mental model of setf was wrong

I realised recently that I’ve been thinking about setf all wrong.

Lisp lets programs define new setf forms for assignment. The most common example is from CLOS, where a class like this:

    (defclass A ()
      ((var
        :accessor a-var)))

will give rise to a class and two functions, a-var to read the value of the var slot on an instance of A, and a setf target used as (setf (a-var instance) 24) to set the var slot of instance.

It’s natural to read that like as executing (a-var instance) to retrieve a location, and setf using this location to assign to. The documentation reinforces this view, talking about “generalised places” as the targets for setf to store things. My mental model was strengthened by idioms like (setf (car pair) 23) to set the car of a pair or list, and (setf (cdr pair) '(1 2 3) to set the cdr. The first argument is a locator expression returning the place to update, and the second argument is the new value to put there.

Natural. But wrong.


The thing I missed is that setf is a macro: it can access the structure of its arguments but not their values. You can’t write code like this:

     (let* ((l (list 1 2 3))
            (h (car l)))
       (setf h 23))

and expect the car of l to be updated, which would make sense if setf were working on a location, because h would be that location. But it isn’t.

What actually happens is that the setf macro looks, at compile time, at the structure of its first (locator) argument, and uses that to dispatch to a method. Using the slot accessor above, the setf form expands to something like:

     (defmethod (setf a-var) (v (a A))
       (setf (slot-value a 'a-var) a))

This is a method with two pieces of selection: specialised on the type of an argument (A), and named with the selector used to within the locator (a-var). It’s definition expands to another setf, this time specialised against slot-value and an instance of standard-object. Specialising on the selector explains why we need that selector to be present syntactically at compile time.

My mistake was thinking that the similarity between access form and setf form was necessary and functional – and it isn’t either. This has some interesting consequences.

The selector is entirely arbitrary

If we don’t like using car to indicate the head of a list – and some people don’t – we could in principle define a new specialisation such as:

     (defmethod (setf head) (v (l list))
       (rplaca l v))

and use it as (setf (head l) 45) even though head isn’t a defined function. All we need is a selector symbol.

There can be more arguments

Ever since I first encountered them I wondered why the lambda lists for new setf specialisations was so strange: the new value and then the arguments – but not the selector – of the place to be updated? Once you get a better mental model, the reason becomes obvious: there can be multiple arguments to the setf locator, possibly actually a variable number, alongside the selector, so we need to be able to find the new value reliably. The easiest way is to put it at the front of the lambda list.

There’s actually a common example of this sitting in plain sight that I’d missed. You access the elements of a Lisp array using the aref function, which takes the array and the index, such as (aref a 23). The corresponding setf form looks like (setf (aref a 23) 0), with the locator taking several arguments like the function. But it isn’t calling the function: it’s decomposing a pattern that looks exactly like the function call for convenience, and which passes several arguments to the specialised method that will look something like:

     (defmethod (setf aref) (v (a array) (i integer))
       ...)

The new value is reliably in the first argument position, with the rest of the locator arguments after it.

You can specialise by value too

Since the setf forms are just methods, you could if you wanted to specialise them on the type of the new value as well as on the locator. As a trivial example:

     (defmethod (setf assign-head) ((v integer) (l list))
       (format t "Assigned an integer ~s" v)
       (setf (car l) v))

     (defmethod (setf assign-head) ((s string) (l list))
       (format t "Assigned a string ~s" s)
       (setf (car l) s))

     (setf (assign-head '(1 2 3)) "zero")
Assigned a string "zero"

Obviously there are better ways to do this, but it’s a good example of the flexibility that comes from setf not really being all that special a form at all: just a creative use of the power of generic functions.

Can we build our own setf-like macros?

Yes: setf is entirely constructable within “ordinary” Lisp.

There are two parts to the construction. Firstly, we need the name of the method that underlies a particular selector.

We can build our own functions with names like this, although not using defun.

     (defvar *weird-name* (make-symbol "(1 2 3)"))

     (setf (symbol-function *weird-name*)
           (lambda (a)
             (print (format nil "We did *weird-name* on ~s" a))))

     (funcall *weird-name* "a string")

"We did *weird-name* on \"a string\""

For setf, the style of name used for the methods implementing the different choices is (setf selector) – a function named by a list – where selector is the symbol at the head of locator list. (Some Lisps construct a symbol from the list elements, rather than using it directly. I’m not sure what, if anything, the Common Lisp language definition says about how this should work.)

For the second part of the construction, setf takes the locator, synthesises the function name symbol using the selector, and calls a generic function with this name, passing the new value and the rest of the locator as arguments.

So to define a new construct our-setf we might do something like:

     (defmacro our-setf (locator new-value)
       (let* ((selector (car locator))
              (our-setf-function-name (make-symbol (format nil "(our-setf ~a)"
                                                           selector))))
         `(apply (symbol-function ,our-setf-function-name)
                 (cons ,new-value ,@(cdr locator)))))

When called as something like (our-setf (head '(1 2 3)) 0) the macro will code to call a method (our-setf head) (as a symbol), passing it (0 '(1 2 3)) as arguments and allowing the machinery of generic functions to determine which method is actually called. We define these methods of the form (our-setf head) and specialise them as required.

(It’s actually a bit more complicated than this because we need to define a generic function for (our-setf head). We have to go backstage and programmatically define the generic function. But the idea remains the same.)


After all this, my mental model of setf is a lot clearer – and, I hope, closer the reality at least. It combines a highly structured use of macros, synthesised function names, and generic functions – and no special machinery at all.

However, there’s some subtlety at play too, not obvious at first acquaintance. We don’t want our synthesised function names to accidentally capture the names of user-supplied code. It’s possible that using a naming style like setf-car would do just this, and a program happens to define a function with this name. But the names setf synthesises are lists, unlikely to be captured accidentally, which lets us define the specialised methods “as normal” even though some of the other parts of the process have to happen backstage.

This shows the power of macros and generic functions. It also shows how deeply the latter are embedded into Lisp. They’re usually thought of as part of CLOS, but they actually have little explicit relationship to class and objects at all, and have been woven all through Lisp to build flexible code structures.

UPDATED 2023-07-30: I incorrectly said originally that one couldn’t use forms like (defun (setf abc) ...): you can, just as with defmethod and defgeneric, and name a function using a list. Thanks to Hacker News contributor phoe-krk for correcting me. I was also slightly loose in my use of specialisation, which I’ve tightened up.

The art of the metaobject protocol

The art of the metaobject protocol

nil

Gregor Kiczales, Jim des Rivières, and Daniel Bobrow. The Art of the Metaobject Protocol. MIT Press. 1991.

What is a meta-object protocol? – or indeed a meta-object? This book is perhaps the clearest exposition of these ideas.

In most modern object-oriented languages an object is an instance of a class. In keeping with using objects throughout, classes are often also objects (or can be thought of as such), but are more informatively thought of as meta-objects that to facilitate the construction of “real” objects. The methods on classes can also be thought of as meta-objects defining the code executed by the objects when invoked.

The defining feature of CLOS is that these meta-objects are all just Lisp objects, but objects that exist “off-stage” (to use this book’s very intuitive metaphor) and largely invisible to a basic user. But they’re as available to a power user as any other objects: the “meta”-ness is a matter of design, not of implementation. The interactions between objects and meta-objects, for example which methods are called when invoked on a particular object, are defined by the meta-object protocol (MOP), which is itself defined in terms of methods on the meta-objects that shadow the objects themselves.

(Meta-object protocol uses a term common in a lot of the earlier object-oriented literature to mean a collection of functions: meta-object API would be a more modern rendering, although the protocol includes the sequencing of API calls and their relationships.)

The goal of MOP programming is to let the programmer extend the programming language towards to application domain, by automating a lot of boilerplate code and providing the structures needed to re-structure or analyse the code the programmer actually needs to write. In this sense it’s a continuation of the idea of macros as powerful and potentially very domain-specific language and compiler extensions. It’s also a continuation of reifying underlying language mechanisms in the language itself where they can be re-specified and re-mixed.

The first part of the book explains MOPs by defining a slightly simplified version of CLOS (“Closette”). It assumes the reader knows some CLOS, for example from Object-oriented programming on Common Lisp: A programmer’s guide to CLOS (or there’s a stand-alone introduction in Appendix A), but it only assumes the knowledge level of a relative newcomer – and the features then defined in Closette are just those parts of CLOS that such a user would actually know and be comfortable with, which is a brilliant piece of pedagogy that simplifies without trivialising. It’s really noticeable that Closette doesn’t need any extensions to Common Lisp: it’s defined directly in the language itself, which shows how powerful the underlying language is. (Full CLOS requires a bit of language support too, at least for efficiency.)

Next come several examples of MOP usage, for example to re-define how classes store their slots, or how to add attributes to slots that can store metadata about their use or could be used to provide higher-level operations. There’s also a long discussion about protocol design and how this has a massive impact on how easy a system is to use for the programmer.

The second part is a manual for the CLOS MOP, which is thorough and useful, but perhaps less exciting than the first part. The Common Lisp package closer-mop provides this API as a portable compatibility layer for use in real programs.

There’s also a discussion of practicalities like where awkward circularities occur and how to break them, which is actually a great example how to do good protocol/API design. In an example of Paul Graham’s dictum that modern languages evolve by mixing Lisp concepts into a different base, MOP ideas appear in lots of other languages, either for real (Smalltalk, at to a lesser extent Python) or just for introspection (Java). Even someone not planning on writing Lisp would benefit from reading this book just to see the ideas in their full generality.

Object-oriented programming on Common Lisp: A programmer’s guide to CLOS

Object-oriented programming on Common Lisp: A programmer’s guide to CLOS

nil

Sonja Keene. Object-Oriented Programming in Common Lisp: A Programmer’s Guide to CLOS. Addison-Wesley. ISBN 0-201-17589-4. 1989.

The definitive practical guide to using the Common Lisp Object System (CLOS). It’s written from a similar perspective to other object-oriented tutorials, which makes it very accessible for those who’ve had experience with something like Java or Python. However, CLOS isn’t just objects in Lisp, and isn’t in any sense just an object-oriented extension. It can take some time to change mindset enough to use it properly, and this book is a great guide to the core differences.

Firstly, it follows a completely different model of how to associate functions with data. Instead CLOS uses “generic” functions, where the exact code called is dispatched dynamically based on the types of any or all parameters: so it’s perfectly possible to have several definitions of the same generic function operating on objects of the same class, but taking arguments of different types. This multiple dispatch is a lot more flexible.

The second point actually follows from this. CLOS’ generic functions can be defined to any Lisp types: in fact they’re not statically associated with classes at all, and can operate on any types (classes or not) across the type hierarchy. This makes it closer to Haskell’s type classes than to Smalltalk’s (or Java’s) virtual methods, which are strongly bound to classes.

Thirdly, CLOS methods can be combined in a range of interesting ways, not simply by overriding previous definitions – and indeed you can define your own if you need to. And like Smalltalk (but unlike Java) CLOS classes have “metaclasses” that can re-define their basic functions. The art of the metaobject protocol is a better source for this level of detail.

The examples in the book delve into these features by means of sensibly-sized challenges that can be used to illustrate both basic design and implementation. and more advanced ideas like re-defining classes on the fly.