Simon Dobson (Posts about software)

New release of epydemic

Simon Dobson — Fri, 29 Nov 2024 16:35:46 GMT

New release of epydemic

It's been a while – about 15 months since v.1.13.1 – but today I finally released v.1.14.1 of epydemic, a library for epidemic (and) other simulation over complex networks.

This release include several bug-fixes and two major changes.

Better Newman-Ziff percolation

The Newman-Ziff algorithm for bond and site percolation is widely used. The version that's been in epydemic for a while emphasise speed over everything. However, in the course of some work by one of my MSc students on how grids of different dimensions respond to random failure, we discovered that we needed to be able to do some more flexible operations. In particular, we wanted to sample things other than just the size of the largest connected component, and wanted to be able to dig into to exactly how the network was deforming.

The problem was that this information wasn't readily available. It was encoded within the algorithm's data structure, but it wasn't being reflected as an evolving network that was easy to get at. So we upgraded the algorithm to build a working copy of the network as it was constructed, so that it could be interrogated by normal networkx operations within the sampling process. This adds some time penalty, but it's acceptable slowdown for the extra capability.

Multiple process instances

epydemic defines different epidemic processes (and indeed non-disease processes like pulse-coupled oscillators). Until now these have been usable alone in a simulation, but not together: one couldn't run two diseases in the same simulation over the same population simultaneously. Doing so is obviously very desirable, especially if you want to explore co-infecting diseases.

Co-infection is a difficult problem. As a first step we've added multiple process instances which can have their own parameters and results – or can share parameters if required. This involves assigning distinct names to each instance, and then optionally using them to decorate parameter/result names.

This is fiddly if done manually, so we also added some methods on the Process class to get and set parameters and results using any instance name on the calling process. For example:

   params = dict()

   # network
   N = 10000
   kmean = 100
   params[ERNetwork.N] = N
   params[ERNetwork.KMEAN] = kmean

   # first infection
   p1 = SIR("Disease1")
   p1.setParameters(params,
                        {SIR.P_INFECT: 0.1,
                         SIR.P_INFECTED: 5.0 / N
                         })

   # second infection
   p2 = SIR("Disease2")
   p2.setParameters(params,
                        {SIR.P_INFECT: 0.3,
                         SIR.P_INFECTED: 5.0 / N
                         })

   # common removal rate
   params[SIR.P_REMOVE] = 0.005

   # run the processes together
   ps = ProcessSequence([p1, p2])
   e = StochasticDynamics(ps, ERNetwork())
   rc = e.set(params).run(fatal=True)

The setParameters call sets the parameters decorated with the name of the process, if it has one. There are other operations for extracting the parameters, and for interacting with experimental results without getting into the details of decoration.

See the project documentation for more details, as well as an updated tutorial and a cookbook recipe for co-infection (which is based around the code above). The Github repo is also available. To upgrade, just run:

   pip install --upgrade epydemic

or delete and re-build any virtual environments.

epydemic hits 100,000 downloads

Simon Dobson — Wed, 08 May 2024 15:21:29 GMT

epydemic hits 100,000 downloads

Downloads of epydemic, my library for epidemic (and other) process simulation on networks (and hopefully other combinatorial structures soon…) recently passed 100,000.

That number comes from the project's PePy page, which tracks downloads from the main PyPi page as used by pip. I can't say whether or not that number is accurate. Quite honestly it's at least 98,000 more than I ever expected, but 100,000 feels like something of a milestone to be pleased about.

epydemic came about because of a lack of standard tooling for doing epidemic simulation. This involves a lot of stochastic simulation, which is quite tricky code to write and to make efficient. Testing the system actually involved us thinking more deeply about the effectiveness of unit testing for stochastic code, which then led to a presentation at UK Systems in 2021 explaining the problems we'd had.

New version of epyc released

Simon Dobson — Mon, 06 Sep 2021 12:50:48 GMT

I released a new version of my epyc experimental management library.

First SleepySketch release

Simon Dobson — Tue, 07 Jan 2014 12:36:14 GMT

Happy 2014! We're particularly happy to be making the first release of the SleepySketch library for writing low-power Arduino sketches. SleepySketch changes the way you write Arduino sketches by letting the library, rather than the main body of the sketch, decide when to run code. The sketch stays asleep as much as possible, with the Arduino placed into a low-power state to preserve battery. This is a first release of SleepySketch, for comments. It provides a sketch framework, a basic sleep manager, and an example "blinkenlights" demonstration to show how the system fits together. Future releases will provide more flexible sleep management and support for component-level power management for common components like Xbee radios. You can download SleepySketch v. 0.1 from here.

Some improvements to SleepySketch

Simon Dobson — Fri, 26 Jul 2013 15:20:38 GMT

It's funny how even early experiences change the way you think about a design. Two minor changes to SleepySketch have been suggested by early testing.

The first issue is obvious: milliseconds are a really inconvenient way to think about timing, especially when you're planning on staying asleep for long periods. A single method in SleepySketch to convert from more programmer-friendly days/hours/minutes/seconds times makes a lot of difference.

The second issue concerns scheduling -- or rather regular scheduling. Most sampling and communication tasks occur on predictable schedules, say every five hours. In an actor framework, that means the actor instance (or another one) has to be re-scheduled after the first has run. We can do this within the definition of the actor, for example using the post() action:

class PeriodicActor : public Actor {
   void post();
   void behaviour();
}

...

void PeriodicActor::post() {
   Sleepy.scheduleIn(this, Sleepy.expandTime(0, 5));
}

(This also demonstrates the expandTime() function to re-schedule after 0 days and 5 hours, incidentally.) Simple, but bad design: we can't re-use PeriodicActor on a different schedule. If we add a variable to keep track of the repeating period, we'd be mixing up "real" behaviour with scheduling; more importantly, we'd have to do that for every actor that wants to run repeatedly.

A better way is to use an actor combinator that takes an actor and a period and creates an actor that runs first re-schedules the actor to run after the given period, and then runs the underlying actor. (We do it this way so that the period isn't affected by the time the actor actually takes to run.)

Actor *a = new RepeatingActor(new SomeActor(), Sleepy.expandTime(0, 5));
Sleepy.scheduleIn(a, Sleepy.expandTime(0, 5))

The RepeatingActor runs the behaviour of SomeActor every 5 hours, and we initially schedule it to run in 5 hours. We can actually encapsulate all of this by adding a method to SleepySketch itself:

Sleepy.scheduleEvery(new SomeActor(), Sleepy.expandTime(0, 5));

to perform the wrapping and initial scheduling automatically.

Simple sleepy sketches can now be created at set-up, by scheduling repeating actors, and we can define the various actors and re-use them in different scheduling situations without complicating their own code.

Sleepy sketches

Simon Dobson — Thu, 25 Jul 2013 12:00:01 GMT

Keeping the microcontroller asleep as much as possible is a key goal for a sensor system, so it makes sense to organise the entire software process around that.

The standard Arduino software model is, well, standard: programs ("sketches") are structured in terms of a setup() function that runs once when the system restarts and a loop() function that is run repeatedly. This suggests that the system spends its time running, which possibly isn't all that desirable: a sensor system typically tries to stay in a low-power mode as much as possible. The easiest way to do this is to provide a programming framework that handles the sleeping, and where the active bits of the program are scheduled automatically.

There are at least two ways to do this. The simplest is a library that lets loop() sleep, either directly or indirectly. This is good for simple programs and not so good for more complicated ones, as it means that loop() encapsulates all the program's logic in a single block. A more modern and compositional approach is to let program fragments request when they want to run somehow, and have a scheduler handle the sleeping, waking up, and execution of those fragments. That lets (for example) one fragment decide at run-time to schedule another

If we adopt this approach,we have to worry about the fact that one fragment might lock-out another. A desktop system might use threads; this is more problematic for a microcontroller, but an alternative is to force all fragments to only execute for a finite amount of time, so that the scheduler always gets control back. This might lead to a fragment not running when it asked (if other fragments were still running), but if we assume that the system spends most of its time asleep anyway, there will be plenty of catch-up time. Doing this results in an actor system where the fragments are actors that are scheduled from an actor queue.

Turning this into code, we get the SleepySketch library: a library for building Arduino sketches that spend most of their time sleeping.

There are a few wrinkles that need to be taken care of for running on a resource-constrained system. Firstly, the number of actors available is fixed at start-up (defaulting to 10), so that we can balance RAM usage.(With only 2k to play with, we need to be careful). Secondly, we use a class to manage the sleeping functionality in different ways: a BusySleeper that uses the normal delay() function (a busy loop) with no power-saving functions, a HeavySleeper that uses the watchdog timer to shut the system down as far as possible, and possibly some other intermediate strategies. Actors are provided by sub-classing the Actor class and providing a behaviour. We also allow pre- and post-behaviour actions to define families of actors, for example sensor observers. We separate the code for an actor from its scheduling.

The standard library uses singleton classes quite a lot, so for example the Serial object represents the USB connection from an Arduino to its host computer and is the target for all methods. We use the same approach and define a singleton, Sleepy

The program structure then loops something like this. If we assume that we've defined an actor class PingActor, then we can do the following:

void setup() {
   Serial.begin(9600);
   Sleepy.begin(new HeavySleeper());

   Sleepy.scheduleIn(new PingActor("Ping!"), 10000);
}

void loop() {
   Sleepy.loop();
}

The setup() code initialises the serial port and the sleepy sketch using a HeavySleeper, and then schedules an actor to run in 10000ms. The loop() code runs the actors while there are actors remaining to schedule. If the PingActor instance just prints its message, then there will be no further actors to execute and the program will end; alternatively the actor could schedule further actors to be run later, and the sketch will pick them up. The sketch will remain asleep for as long as possible (probably for over 9s between start-up and the first ping), allowing for some fairly significant power saving.

This is a first design, now just about working. It's still not as easy as it could be, however, and needs some testing to make sure that the power savings do actually materialise.

The xbee-arduino library

Simon Dobson — Wed, 03 Jul 2013 07:30:22 GMT

A library for using XBee radios with Arduinos.

The XBee radio operates in two modes: transparent or text-based, and API or binary-based. The latter (API mode) is generally considered more suitable for computer-to-computer interactions, as it's faster and simpler for computers to manipulate. However, using an XBee in this mode requires additional software.

The xbee-arduino library provides Arduino functions to access the API mode functionality of the various XBee radio modules. The library is quite low-level, but does provide access to all the necessary functions like issuing AT commands to control the modem and sensing and receiving packets of data to other radios in the mesh network.

To use the library you download the latest version from the web page and unpack it into the libraries/ directory of your Arduino IDE. You also need to make sure that the radios you use have the API function set installed using X-CTU, as the library only makes sense for radios in API mode. You also have to set the "AP" parameter to 2 when writing the firmware.

XBee firmware management with X-CTU

Simon Dobson — Tue, 02 Jul 2013 08:00:34 GMT

Managing Digi's XBee radio modules requires using their X-CTU package to upload the correct firmware. In this post we explain how.

This post is slightly depressing: not because it's on an unhappy topic, but because the effort and software needed to manage XBee radios is so complex to set up. In many ways this is just a function of the good design of the XBee: it's so minimal in terms of footprint and power consumption at run-time that it offloads a lot of work to external tools (and the user) at system build-time. But it's still a lot of work to get such a small piece of kit running.

X-CTU is intended to upload firmware to XBee radio modules. This is needed to change the firmware between router and co-ordinator of the Zigbee mesh network, and between the different protocol variants that the XBee radios can support.

One limitation of X-CTU is that it only works on Windows: if you're running Linux, X-CTU will run under Wine. You can download the latest X-CTU from Digi's X-CTU page; alternatively, there's a version installed on the Citizen Sensing VM.

To use X-CTU you need to connect your XBee module to your computer. The easiest way to do this is using an XBee USB breakout board, which provides an XBee socket and a USB socket. Insert the radio into the board, plug in a USB cable, and plug the other end into a USB socket. The light on the breakout board will then come on (see photograph above).

You next need to start up X-CTU and tell it where the XBee is. It hangs off a Windows COM port, and X-CTU will typically find it automatically. You should then be able to press the "Test/Query" button, and X-CTU will interrogate the XBee and display a small window showing some information about it, as shown in the screenshot on the right: the details don't matter, but this shows that the XBee is talking to the computer properly.

Assuming everything is now working correctly, the next step is to decide what firmware to download to the radio. Click on the "Modem configuration" tab, and then click the "Read" button: this reads the firmware that's on the XBee at the moment, and puts the details into the window. Typically this results in a display like that shown on the left. The important things to notice are the two drop-down boxes labelled "Function Set" and "Version". The function set is the description of the firmware, in which case indicating that the XBee is running Zigbee router firmware that responds to AT commands (more on this below).

To download a new firmware, we then select the function set and version we want to use. Suppose we want to make this XBee into the mesh co-ordinator. We change the function set to "Zigbee Coordinator AT" (keeping with Zigbee and the AT command set) in "Function set" the drop-down, then select the most recent version of this function set from the "Version" drop-down. (Versions are identified by hex numbers: the most recent in the screenshot right is "20A7", that being the highest hex number. Unfortunately X-CTU orders the numbers alphabetically, not in hex-numeric order.) Pressing "Write" will update the radio's firmware, and the radio is then ready for use as a co-ordinator.

If you look through the list of function sets, there will be quite a few options, including protocols other than Zigbee. These probably aren't worth too much exploration, but you'll also notice that there are Zigbee AT and API function sets corresponding to the two modes (transparent and API) that the XBee can support. Be sure to select the correct one for your application.

That's it: the radio is now ready for use.

Advanced use: setting optional parameters

There's one more thing that X-CTU can be used for: it can set parameters to the firmware function set, and this is sometimes important when using the radios.

When you've read the firmware from a radio, the main part of the X-CTU window contains a hierarchy of folders and cryptic values, for example "(4) PL - Power level". These are parameters that can be changed to modify the detailed behaviour of the radio. Some can also be set using AT commands. The example we used sets the radio's power level to 4. If you click on this, it will show a drop-down box giving other options. You might, for some applications, choose to reduce the radio power to 1 ("low") to save batteries. If you choose this and then write the firmware to the radio, the module will use this power setting.

In the example shown on the left, we're changing the AP parameter ("API enable") to 2, which is needed for the xbee-arduino library to work properly. If we now write the firmware (with the Zigbee co-ordinator API function set selected as shown), the radio will be ready for use.