Let’s start with design. Lisp is over half a century old. I’d argue it was exceptionally well-designed – when it was designed. It lacks most modern advances in types because … well, they didn’t exist, many of them arose as solutions to perceived problems in Lisp (and Fortran), and many of those “solutions” still aren’t universally accepted, such as static typing itself.

What we’ve actually learned is that many aspects of programming lack any really universal solutions. If static typing were such an obvious and unarguable route to efficiency and quality, all new software would be being written in Haskell.

And the lack of modern types isn’t really as clear-cut as it appears. The argument about the lack of features in Lisp also ignores the presence of other features that are absent from almost all other languages.

Lisp’s numeric types are surprisingly flexible. Indeed, Common Lisp is still, in the 21st century, just about the only language in which one can write modern crypto algorithms like Diffie-Hellman key exchange without recourse to additional libraries, because it has arbitrary-precision integer arithmetic built-in to the standard operators. It also has rational numbers, so no loss of precision on division either.

The Common Lisp Object System (CLOS) is vastly more flexible than any modern object-oriented language. Sub-class methods can specify their relationship with the methods they override, such as being called after or just filtering the return values. Methods themselves are multiple-dispatch and so can be selected based on the types of their arguments as well as their target. The basic mechanisms can be overridden or extended using a meta-object protocol.

Then there are macros. It’s easy to underestimate these: after all, C has macros, doesn’t it? Well, yes – and no. A C macro is a function from strings to strings that can do literal string substitution of its arguments. A Lisp macro is a function from code to code that can perform arbitrary computation. They’re really not the same things at all, and it’s misleading that the same word is used for both. (C++ templates are a closer analogy, but still limited in comparison.)

The complaints about car and cdr are long established: they were originally derived from machine-language instructions on the IBM 704 that was used for the first Lisp implementations. They’re a terrible hold-over from that terrible decision … aren’t they?

Well, yes – and no. Of course they’re terrible in one sense. But car and cdr are basically nouns as far as Lisp programmers are concerned. One could replace them with more modern usages like head and tail (and indeed many Lisps define these using macros).

But it’s important to remember that even “head” and “tail” are analogies, sanctified by familiarity in the computer science literature but still inexplicable to anyone outside. (If you doubt that, try explaining to someone who isn’t a programmer that a shopping list has a “head” consisting of the first entry, and a “tail” consisting of another, shorter, shopping list, is “in fact” a recursive type, and you have to acquire each item of shopping sequentially by working your way down the list from the head.) car and cdr are artificial nouns, and cons is an artificial verb – but really no more artificial that head, tail, and append, their rough equivalents in other languages.

One can argue that the persistence of car and cdr drives the persistence of compounds like caaddr. But those are unnecessary and seldom used: barely anyone would mind if they were removed.

The suggestion that Lisp has hardware holdovers that should be removed also neglects these holdovers in other languages.

As an example, check the definition of std::memcpy in C++. It doesn’t work with overlapping memory areas. Why is that? – why is it so fast, but so dangerous? Does it relate to underlying machine features, such as machine code move instructions on particular machines with particular restrictions? Doesn’t this introduce the risk of security flaws like buffer overruns?

Languages with more abstracted machine models don’t have these issues. I struggle to think of how one could even introduce the concept of a buffer overrun into Lisp, other than by using some external raw-memory-access library: the language itself is immune, as far as I know.

For the sake of argument, let’s turn the argument around and ask: give that early Lisps had proper macros, arbitrary-precision integers, and so on, why did these features disappear from what we now consider to be “the mainstream” of programming language design?

Lisp’s designers had a goal of building a powerful machine in which to think: indeed, they intended it to eventually have its own hardware designed specifically for it to run on. They therefore didn’t buy into the necessity of immediate performance, and as their applications were largely symbolic AI they didn’t need numerical performance at all. They chose instead to create high-level constructs even if these couldn’t be compiled efficiently, and explored using these to create more code as they identified more and more abstract patterns whose details could be automated away. (Paul Graham has a great essay on this.)

Other language designers had other priorities. Often they needed to do numerical simulation, and needed both performance and scale. So they chose a different design pathway, emphasising efficient compilation to the hardware they had available, and made the compromises needed to get it. These have persisted, and that’s why we have languages with fixed-width integers scaled to fit into a single machine register, and compilers that generate – but don’t directly execute – the code of programs, which limits our ability to abstract and automate code generation without recourse to complicated external tools.

It’s interesting to explore these choices. They’re at one level “just” historical: accidents that shaped the present. But at another level they’re still very much present in the hardware and software landscape we inhabit. I think it’s important that we remind ourselves, continuously, that much of that landscape is a choice, not a given, and one we can question and change as we wish.

The old-school approach to email involves several elements, each potentially provided by a different tool:

• a client program or mail user agent (MUA) that presents email to you and lets you search, delete, store, etc;
• a retrieval program or mail delivery agent (MDA) that retrieves mail from the providers and manages local email directories
• a sending program or mail transfer agent (MTA) that takes; and locally-created messages and transfer them to their intended recipients.

Modern GUI email clients like Thunderbird typically wrap-up all three services into one program that’s easier to deploy and manage, but that therefore forces certain choices on the user. By reverting to the older architecture we regain flexibility and choice, at the expense of making our lives harder.

All these tools need to authenticate against other services. Traditionally this used usernames and passwords, which are clearly inadequate for the modern web. Instead we need a system based around stronger encryption.

OAUTH2 is a an authorisation delegation protocol that lets a site grant access to someone who’s authenticated against another, without getting sight of their credentials. The typical use case is for a web site to allow users to sign-in using social media services like Facebook or Google, which reduces the number of passwords a user needs to remember or manage – and, totally incidentally I’m sure, improves the social media services’ ability to track users’ activities across the web.

In our case, the OAUTH2 “flow” interacts with the authentication provider and acquires a bearer token that can then be presented to authorise access to the various email services.

In outline the solution is as follows:

1. Install mbsync as MDA
2. Set up OAUTH2 authentication for Office365
3. Use the to authenticate mbsync against Office365 to allow retrieval
4. Install msmtp as MTA, using the same authentication scheme
5. Install mu4e as MUA, since I want to read my email from inside Emacs

Under Arch Linux we need the isync package for synchronisation and the cyrus-sasl-xoauth2 provider for OAUTH authentication.

   sudo pacman -S isync
   yay -S cyrus-sasl-xoauth2

The same packages are available for other distros under similar names. Note that the actual synchronisation tool is called mbsync, even though the package that contains it is called isync.

We want to use OAUTH2 to authenticate an IMAP transaction, so that no additional passwords are needed. To this we need a script to manage the OAUTH2 flow.

Weirdly for an operation that’s becoming so common on the web, there doesn’t seem to be a package that offers OAUTH2 from the command line. However, there is a script that does it that’s included as an example with the mutt MUA, and we can use that. It can be found (in Arch) in the mutt package.

   sudo pacman -S mutt
   cp /usr/share/doc/mutt/samples/mutt_oauth2.py .

This puts a copy of the script into the current directory, which we can then edit in two ways:

• add the internal application identification and client secrets for accessing Office365; and
• set up the security for the OAUTH2 access tokens when they’re downloaded and held locally.

The client secret and app id need to be “proper”, in the sense that Office365 knows about them – but weirdly they don’t have to be related to your email domain or cloud tenancy. It’s perfectly fine to use credentials available in the public domain, for example those of Thunderbird:

   AppID = "08162f7c-0fd2-4200-a84a-f25a4db0b584"
   ClientSecret = "TxRBilcHdC6WGBee]fs?QR:SJ8nI[g82"

(I got these from here, but other than that have no idea where they come from: they’re not the same as those in the Thunderbird source code, as far as I can tell.)

The mutt_oauth2.py script stores the tokens it manages in a gpg-encrypted file. You therefore need to provide your gpg keypair identification, and I’m assuming anyone wanting to get local email has one of those! Mine is “simoninireland”.

   GPGKey = "simoninireland"

I edited the file to look like this, with some details elided:

   MSAppID = "08162f7c-0fd2-4200-a84a-f25a4db0b584"
   MSClientSecret = "TxRBilcHdC6WGBee]fs?QR:SJ8nI[g82"
   GPGKey = "simoninireland"

   ENCRYPTION_PIPE = ['gpg', '--encrypt', '--recipient', GPGKey]
   DECRYPTION_PIPE = ['gpg', '--decrypt']

   registrations = {
       'google': {
           ...
           'client_id': '',
           'client_secret': '',
       },
       'microsoft': {
           ...
           'client_id': MSAppID,
           'client_secret': MSClientSecret,
       },
   }

Put resulting script into /usr/local/bin and make it executable. Then run it in “authorisation” mode. The token file can go anywhere: I put it in the directory used by pass to allow for an alternative access route:

   mutt_oauth2.py -t .password-store/email/work.gpg --authorize

This will ask some questions:

• we want “microsoft” authentication
• and a “localhostauthcode” flow
• enter your email address (the actual user, not any alias)

and it prints out a URL to copy into a browser to authenticate against Office365’s web interface. In my case this involved interacting with the university’s single sign-on and two-factor authentication (2FA) system. Doing this successfully put the necessary OAUTH2 tokens, encrypted, into the specified file. Running:

   mutt_oauth2.py -t .password-store/email/work.gpg

will output the token, refreshing it automatically if it’s expired. This may ask for the GPG key’s passphrase, if it has one, and if it’s not available from a local key agent.

(All this security means that the bearer tokens are stored encryoted at rest. It’s a little inconvenient, though, as it means you need to enter a gpg passphrase periodically, and makes it hard to run mbsync in a cron job. This is fine if, like me, your level of security paranoia is such that you accept the minor inconvenience in exchange for not having plain-text access tokens lying around; on the other hand, you may decide that using, for example, a machine with full-disc encryption is secure enough, in which case you need to edit the ENCRYPTION_PIPE and DECRYPTION_PIPE commands in the script to not do encryption: they can basically just use cat to store and retrieve the token information.)

We now have OAUTH2 tokens for accessing Office365, which we can integrate with our MDA. mbsync has four main concepts:

• Accounts, typically using IMAP
• IMAP message stores, which are remote
• Maildir stores, which are local
• Channels, which tie local and remote together

Maildir is a file format for storing email in a directory structure, and is a long-running standard that’s supported by lots of tools. A maildir is typically presented in the MUA to a user as a folder, and represented to the MDA as a directory.

For Office365 we have:

   IMAPAccount work
   Host outlook.office365.com
   Port 993
   User <<work-email>>
   PassCmd "mutt_oauth2.py -t ~/.password-store/email/work.gpg"
   AuthMechs XOAUTH2
   SSLType IMAPS

   IMAPStore work-remote
   Account work

   MaildirStore work-local
   Subfolders Verbatim
   Path ~/Maildir/Work/
   Inbox ~/Maildir/Work/Inbox

   Channel Work
   Far :work-remote:
   Near :work-local:
   Patterns * !"Conversation History" !Calendar !Archive !Archives !Clutter !Drafts
   Create Both
   Expunge Both
   SyncState *

(See the mbsync man pages for the details of its configuration. <<work-email>> should be a a proper username, not an alias.) For our purposes the important line is the PassCmd that calls our edited script to retrieve the OAUTH2 bearer token. Email will be downloaded into a maildir tree rooted at ~/Maildir/Work: you need to create this before sync-ing.

   mkdir -p ~/Maildir/Work

For a full sync of all maildirs just run:

   mbsync -a

That can be time-consuming, as all the maildirs (i.e., folders) have to be visited – and I have several hundred. A faster option is to normally just look at (for example) the inbox:

   mbsync Work:INBOX

This will ignore everything else, which means they’ll drift – but can be re-sync’ed periodically by running a full sync. One could also set up a cron job to do a full sync early every morning, for example, as long as the access token was held unencrypted (see above).

You’ll almost certainly now want to index your newly-downloaded trove of messages. There are two common tools for this mu and notmuch. Both do basically the same job of maintaining a structured and full-text index of messages that can be queried by an appropriate MUA. I chose mu, for no particular reason: some people swear by notmuch, which is based on extensive tagging of messages and so might be more familiar to Gmail users.

To install mu, we first grab the package:

   pacman -S mu

We then initialise the index by running the indexer over the maildir. If we also provide our own email address (or more than one) it knows to index these differently.

   mu init -m ~/Maildir --my-address=<<work-email>>

All of the above sets up the MDA to get mail: we now need to be able to send mail. Fortunately we’ve already done most of the hard work needed to get this working.

We need a local MTA, for which I chose msmtp. It understands OAUTH2 natively. Installation in Arch is easy:

   sudo pacman -S msmtp

It needs to be pointed at the Office365 SMTP server and provided with the OAUTH2 tokens, which are the same as we used above:

   defaults
   auth           on
   tls            on
   tls_starttls   on
   tls_trust_file /etc/ssl/certs/ca-certificates.crt
   logfile        ~/.msmtp.log

   account        work
   host           smtp.office365.com
   port           587
   auth           xoauth2
   user           <<work-email>>
   from           <<work-email>>
   passwordeval   "mutt_oauth2.py -t ~/.password-store/email/work.gpg"

   account default : work

Again, see the msmtp man pages for the details of this, and replace <<work-email>> as appropriate: the only interesting part from our current perspective is that the passwordeval line calls exactly the same script as we used above.

Finally we’re ready to read email. I’ll leave this to you: there are lots of text-based email clients around, notably mutt that we encountered earlier. There’s also mu4e for reading email in Emacs, making use of the mu index; and notmuch also has an Emacs interface.

I use mu4e. There’s a lot of documentation on the web for setting this up, all of which applies immediately to our new set-up: the MUA is entirely independent of the MDA and MTA, and simply needs to be pointed at the right directories and accounts.

Gmail lets one use “app passwords” for accessing using IMAP, but also supports OAUTH2, which is obviously more secure. The same approach as above works for Gmail too. The initial credentials are:

   GAppID = '406964657835-aq8lmia8j95dhl1a2bvharmfk3t1hgqj.apps.googleusercontent.com'
   GClientSecret = 'kSmqreRr0qwBWJgbf5Y-PjSU'

(Same source as above.) Edit these into the script and change the entries in the config files to call it to authenticate with an appropriate store, for example:

   mutt_oauth2.py -t .password-store/email/personal.gpg --authorize

and similarly in the configurations of mbsync and msmtp.

If you’re still with me: congratulations, but you must really want to read your email old-school!

For me, this has completely changed my relationship with email in ways I didn’t expect. Using Emacs means typically not having the client visible all the time, which reduces the temptation to check all the time. Instead I can adopt a more structured approach and only check my email when I want to, which often means only three or four times a day. It’s also made email easier to manage, for example by adding hyperlinks in my to-do list straight to messages that need attention, and adding some integrations with org mode to simplify email processing. Those are matters for another time, though.

There are many resources on using mbsync, mu, mu4e, and the rest on the web. I found these covered all the topics in great detail, with the exception of the OAUTH2 integration I’ve detailed here. In particular I’d like to acknowledge the following:

• Growing pains, Casper van Elteren’s mu4e setup that pointed me in the right direction
• Microsoft OAuth authentication with mu4e in Emacs, which uses offlineimap, a different MDA, and needs closer integration with Microsoft Azure that often isn’t possible for corporate email
• Configure Mutt to work with OAuth 2.0 that focuses on using mutt and Gmail