Lisp macros versus Rust macros

Lisp macros versus Rust macros

I was talking with one of my colleagues the other day about programming languages, and we ended up comparing macros in Rust and Lisp.

Rust has a couple of couple of different kinds of macros: declarative macros that pattern-match on arguments to emit code; and procedural macros that perform more general code-to-code transformations. Lisp has only one kind that operates from code to code.

Both approaches are far more powerful than the macros in C and C++, which are basically just string expanders. Indeed, one definition of macroprogramming is that it’s writing code that returns code, and there’s a reasonable argument that C’s “macros” are programs that return strings and therefore aren’t macros at all. But that’s just bring pedantic.

The Rust operations seem quite awkward, at least from a Lisp perspective. They’re invoked in a way that’s syntactically different to ordinary code, so it’s always possible to see in the source code where procedural code generation is occurring. Perhaps that’s not an entirely bad thing, as it makes it obvious when compile-time computation occurs – although one might also argue that a true language extension or DSL should be so seamless that you don’t need to see it.

I think a more basic difference is in how Rust needs to handle code-type arguments. A macro is a function from code to code, so it needs to represent its code arguments in a way that the macros (which is also code) can manipulate. Lisp’s homoiconicity makes this trivial: code is a list, just like non-code, and can ba manipulated as such. Rust doesn’t have this, so code needs to be passed to macros as a token stream that’s been parsed from the program text. That’s a reasonable solution to the problem, but it does mean that to write macros you need to understand how Rust is tokenised. You also get a token stream, not an abstract syntax tree (AST), which means that manipulating complex code is more difficult: essentially you need to re-create as much of the AST as you need and traverse it within the macro body. There’s a standard library that does this for Rust’s own syntax, which simplifies matters somewhat but still means that writing macros exposes the programmer to the underlying representations. Hopefully they won’t change, as that would break a lot of macros.

By contrast, Lisp macros only require an understanding of Lisp itself, not of its internals, and can operate on the entire detailed structure of the code arguments. It’s a striking example of the power of homoiconicity.

An approach closer to that of Rust is also available, in Common Lisp anyway, in the form of reader macros that modify the Lisp reader to allow access to the character stream as the source code is being read. I think I’ve only ever encountered read macros for providing new styles of literals, or variants of strings that benefit from being treated slightly differently at read-time: they’re an unusual use case, anyway, and Lisp makes the more usual case of macros manipulating Lisp code a lot simpler, without exposing the programmer to parsing.

I suspect the main difference between the two languages’ approaches is that macros are additional to Rust but inherent to Lisp. None of the core of Rust uses macros: they’re for extensions. By contrast, even common operations like defun in Lisp are actually macros that expand to the simpler core operations. This perhaps explains the Rust designers’ decision to make macros syntactically distinct.

tags
lisp programming rust macroprogramming

Processing MicroMoth recordings offline

Processing MicroMoth recordings offline

The uMoth generates .wav files, uncompressed waveforms of what it records. These need to be processed to identify any bird calls within them.

This function is integrated in BirdNET-Pi, which does recording and classification, and provides a web GUI. With the uMoths we need to provide classification as part of a data processing pipeline. We can however make direct use of the classifier “brain” within BirdNET-PI, which is unsurprisingly called BirdNET-Analyzer.

Installation

I’m working on a 16-core Intel Core i7@3.8GHz running Arch Linux.

First we clone the BirdNET-Analyzer repo. This takes a long time as it includes the ML models, some of which are 40MB or more.

    git clone https://github.com/kahst/BirdNET-Analyzer.git
    cd BirdNET-Analyzer

The repo includes a Docker file that we can use to build the analyser in a container.

    docker build .

The container setup is quite basic and is probably intended for testing rather than production, but it gives a usable system that could then be embedded into something more usable. The core of the system is the analyze.py script.

Analysing some data (AKA identifying some birds!)

The container as defined looks into its /example directory for waveforms and analyses them, generating text file for each sample. The easiest way to get it to analyse captured data is to mount a data directory of files onto this mount point (thereby shadowing the example waveform provided).

There are various parameters that configure the classifier. I copied the defaults I was using with BirdNET-Pi, only accepting classifications at or above 0.7 confidence.

    docker run -v /var/run/media/sd80/DATA:/example birdnet-analyzer analyze.py --rtype=csv --min_conf=0.7 --sensitivity=1.25

This crunches through all the files (982 of them from my first run) and generates a CSV file for each. An example is:

Start (s) End (s) Scientific name Common name Confidence
6.0 9.0 Corvus monedula Eurasian Jackdaw 0.9360
9.0 12.0 Corvus monedula Eurasian Jackdaw 0.8472
12.0 15.0 Corvus monedula Eurasian Jackdaw 0.8681
15.0 18.0 Corvus monedula Eurasian Jackdaw 0.8677
24.0 27.0 Columba palumbus Common Wood-Pigeon 0.9198
27.0 30.0 Columba palumbus Common Wood-Pigeon 0.7716
45.0 48.0 Corvus monedula Eurasian Jackdaw 0.8023
48.0 51.0 Corvus monedula Eurasian Jackdaw 0.7696

Those are entirely credible identifications. The start- and end-point offsets allow rough location within the recording. (BirdNET segments the recordings into 3s chunks for analysis.)

This is clearly not as straightforward as BirdNET-Pi, nor as immediately satisfying. But it does scale to analysing lots of data (and could be made to do so even better, with a better front-end to the container), which is important for any large-scale deployment.

Deploying a MicroMoth

Deploying a MicroMoth

The MicroMoth (or uMoth) from Open Acoustic Devices is the same as their better-known AudioMoth recorder but with a significantly smaller footprint. It’s just a traditional recorder or data-logger, with now on-board analysis and no wireless connectivity. I got hold of some to use in a larger project we’re thinking about running, and they’re not kidding about the “micro” part.

nil

The uMoth uses the same software as the AudioMoth, and therefore the same configuration app available from the apps page – for 64-bit Linux in my case. It downloads as a .appimage file, which is simply a self-contained archive. It needed to be marked as executable, and then ran directly from a double-click. (The page suggests that there may be some extra steps for some Linux distros: there weren’t for Arch.)

I then followed the configuration guide. The time is set automatically from the computer’s clock when you configure the device.

For testing I chose two recording periods, 0400–0800 and 1400–1600.

nil

As shown this will, with the default 48KHz sampling, generate about 2GB of data per day and use about 70mAh of energy. For my tests I just hung the device out of the window on a USB tether for power: it works fine drawing power from the USB rather than from the battery connector.

nil

This turned out not to record anything, because the time is lost if the power is disconnected, even though the configuration is retained. (The manual does actually say this, with a suitably close reading. It could be clearer.) There’s a smartphone app that can reset the time once the device is in the field and powered-up, though, by making an audio chime that encodes the current time and location in a way the board can understand. Flashing the device with the “Always require acoustic chime on switching to CUSTOM” makes it wait after power is applied until its time is set.

The red LED flashes when the device is recording. The green LED flashes when the device is waiting for a recording period to start. The red LED stays lit while the time is unset.

Pascal Costanza’s highly opinionated guide to Lisp

Pascal Costanza’s highly opinionated guide to Lisp

Pascal Costanza’s Highly Opinionated Guide to Lisp

Part introduction, part paean to the language’s power, part study guide, while dipping into an eclectically-chosen subset of Lisp features that really illustrate what makes it different.

A road to Common Lisp

A road to Common Lisp

A Road to Common Lisp

This a really brief, yet really interesting, approach to introducing Lisp to someone. Interesting because it covers all the usual ground, but also has copious pointers to other material slightly-beyond-introductory (“Where to go from here”). It also links to material that’s essential to modern practice, such as Lisp packages and systems, and the essential “standard libraries” such as Alexandria, Bordeaux, CL-PPCRE, usocket, and the like: the things that are needed in practice and which in other languages would probably be built-in and included directly in an introduction.