The Sapere project (of which I’m a member) is running a workshop at SASO 2012 in Lyons on new ways to design service ecosystems. Emerging distributed computing scenarios (mobile, pervasive, and social) are characterised by intrinsic openness, decentralisation, and dynamics. According, the effective deployment and execution of distributed services and applications calls for open service  frameworks promoting situated and self-adaptive behaviours, and supporting diversity in services and long-term evolvability. This suggests adopting nature-inspired and/or socially-inspired approaches, in which services are modelled and deployed as autonomous individuals in an ecosystem of other services, data sources, and pervasive devices. Accordingly, the self-organizing interactions patterns among components and the resulting emerging dynamics of the system, as those of natural systems or of social systems, can inherently exhibit effective properties of self-adaptivity and evolvability. Although many initiatives (like those named upon digital/business service ecosystems) recognise that the complexity of modern service systems is comparable to that of natural ecosystems, the idea that nature – other than a mean to metaphorically characterise their complexity – can become the source of inspiration for their actual modelling and implementation is only starting being metabolised. The goal of this workshop is to bring together researchers and practitioners, with the aims of unfolding the many challenges related to the modelling, design and implementation of adaptive service ecosystems in natural and social terms, and identifying promising approaches and solutions. Topics of interest include, but are not limited to:

• Software architectures for  emergent distributed systems.
• Bio-inspired self-organising patterns design patterns.
• Coordination models and languages.
• Middleware platforms
• Dynamic services composition.
• Adaptive coordination models and patterns
• Self-organisation and coordination
• Coordination in systems of feedback loops
• Middleware for adaptive coordination
• Multiagent systems
• Methodologies for adaptive and self-organising system engineering

All accepted papers will be published by IEEE Xplore following the workshop. Submission deadline is 4 July 2012. For more details please see the workshop web site. UPDATE: Fixed typo in the submission deadline

The layering of abstractions has served us well, but it’s now generating the sorts of complexity it was designed to solve. Time for a re-think? Anyone who’s built a large piece of software knows that much of the effort is in managing the complexity of the project: which other software a piece of code relies on, how to keep the various aspects separate, how to manage changes and upgrades, and so on. This isn’t something that’s got easier over time: it has for a given code size and style, as we’ve understood build processes and dependency management better; but the code sizes have relentlessly increased to compensate for our improved understanding; and modern practices don’t make life any easier. Downloaded code, dynamic modules and classes, client-server and the like all generate their own intrinsic complexity. One of the biggest sources of complexity is the use of multiple applications, especially in enterprise systems. A typical e-commerce system, for example, will make use of a web server to present pages (which themselves might contain embedded JavaScript for client-side processing), a database to track orders and inventory, a procurement system to fulfil orders, and possibly a supply-chain management system to order new inventory. That’s the application. Then there’ll be the operating system, a logging facility, a facilities management system, and a load of administrative tools and scripts. And the operating system may itself be virtualised and running as a guest within another, host operating system and hypervisor, which needs its own toolset. The interactions between these tools can be mind-boggling. Someone once asked: who knows how to work the Apache web server? It sounds like a simple question — any competent web manager? the main developers? — but the sting in the tail is that Apache is very configurable: so configurable, in fact, that it’s pretty much impossible to work out what a given combination of options will do (or, conversely, what combination of options to use to achieve a given effect). The interactions are just too complicated, and the web abounds with examples where interactions between (for example) the thread pool size, the operating system block size, and the Java virtual machine parameters conspire to crash a system that looks like it should be working fine. If you can’t work one server properly — one component of the system — what hope is there to get a complete system humming along? Al Dearle and I have been talking about this for a while. The basic issue seems to be an interaction between decomposition and dependency. In other words, the complexity comes at the “seams” between the various sub-systems, and is magnified the more configurable the components on either side of the seam are. This is important, because systems are becoming more finely decomposed: the move to component software, software-as-a-service and the like all increase the number of seams. Al’s image of this is that modern systems are like Russian dolls, where each supposedly independent component contains more components that influence the size and complexity of the component containing them. You can only simplify any individual piece so far, because it depends on so many other pieces. Actually a lot of the seams are now unnecessary anyway. Going back to the e-commerce example, the operating system goes to great pains to provide a process abstraction to keep the components separate — to stop faults in the database affecting the web server, for example. Historically this made perfect sense and prevented a single faulty process in a time-sharing system affecting the processes of other users. Nowadays, however, it makes considerably less sense, for a number of reasons. Firstly, all the components are owned by a single nominal user (although there are still good reasons for separating the root user from the application user), so the security concerns are less pronounced. Secondly, all the components depend on each other, so a crash in the database will effectively terminate the web server anyway. (We’re simplifying, but you get the idea.) Finally, there’s a good chance that the web server, database and so on are each running in their own virtual machine, so there’s only one “real” process per machine (plus all the supporting processes). The operating system is offering protection that isn’t needed, because it’s being provided (again) by the hypervisor running the virtual machines and perhaps (again) by the host operating system(s) involved. We also tend to build very flexible components (like Apache), which can deal with multiple simultaneous connections, keep users separate, allow modules to be loaded and unloaded dynamically — behave like small operating systems, in other words, replicating the OS functionality again at application level. This is despite the fact that, in enterprise configurations, you’ll probably know in advance the modules to be loaded and have a single user (or small user population) and fixed set of interactions: the flexibility makes the component more complex for no net gain during operation. Although it might simplify configuration and evolution slightly, there are often other mechanisms for this: in a cloud environment one can spin-up a replacement system in an evolved state and then swap the set of VMs over cleanly. It’s easy to think that this makes no difference for modern machines, but that’s probably not the case. All these layers still need to be resourced; more importantly, they still need to be managed, maintained and secured, which take time to do well — with a result that they typically get done badly (if at all). Can we do anything about it? One thought is that the decomposition that makes thinking about systems and programming easier makes executing those systems more complex and fragile. In many cases, once the system is configured appropriately, flexibility becomes an enemy: it’ll often be too complicated to re-configure or optimise in a live environment anyway. There may be a reason to have Russian dolls when designing a system, but once designed it’s better to make each doll solid to remove the possibility of then opening-up and falling apart. So it’s not decomposition that’s the issue, it’s decomposition manifested at run-time. When we add new abstractions to systems, we typically add them in the form of components or libraries that can be called from other components. These components are often general, with lots of parameters and working with multiple clients — sound familiar? This is all good for the component-writer, as it lets the same code be re-used: but it bloats each system that uses the component, adding complexity and interactions. So one thought for tackling complexity is to change where decomposition manifests itself. If instead of placing new functions in the run-time system, we placed it into the compiler used to build the run-time, we could use compilation techniques to optimise-out the unnecessary functionality so that what results is optimised for the configuration that it’s actually being placed in, rather than being general enough to represent any configuration. There’s substantial work on these ideas in the fields of staged compilation and partial evaluation (for example MetaOCaml, Template Haskell, Flask and the like): the flexibility is manifested at compile-time as compile-time abstractions, that in the course of compilation are removed and replaced with inflexible — but more efficient and potentially more dependable — specialised code. Think taking the source code for Linux, Apache and MySQL, accelerating them together at high speed, and getting out a single program that’d run on a bare machine, had nothing it didn’t actually need, and had all the options for the various (conceptual) sub-systems set correctly to work together. Don’t believe it’s possible? Neither do I. There’s too much code and especially too much legacy code for this to work at enterprise (or even desktop) level. However, for embedded systems and sensor networks it’s a different story. For these systems, every extra abstraction that makes the programmer’s life easier is a menace if it increases the code size hitting the metal: there just isn’t the memory. But there also isn’t the legacy code base, and there is a crying need for better abstractions. So an approach to the Russian dolls that moves the abstractions out of the run-time and into the languages and compilers might work, and might considerably improve the robustness and ease of use for many systems we need to develop. It also works well with modern language technology, and with other trends like ever-more-specialised middleware that remove bloat and overhead at the cost of generality. Keeping the former and the latter seems like a worthwhile goal.

We have five more PhD scholarships tenable from September, for students interested in any of our areas of research. The details are available here. Please note that I’m not taking on a PhD student this year, so don’t bother applying to work with me.

Beware of books bearing illnesses. I recently read Hindenburg: the wooden titan, John Wheeler-Bennet’s biography of the presidency of the man who ushered Hitler into power. It’s an interesting book, written just after Hindenburg’s death and so before the second world war and the realisation of exactly what had been brought forth. You can also find contemporary book reviews online which are likewise fascinating in what they don’t know. It really brings home AJP Taylor’s comment (in The struggle for mastery in Europe) that that hardest thing about the study of history to to remember that events now long in the past once lay in the future. What was unexpectedly fascinating was the bookplate in the front of the volume I borrowed from the university library, which was published in 1937. (I think it’s a first edition.) The top half is fairly standard, but the bottom part describes something you wouldn’t expect to find in a library book: “A person shall not return any public library book which he knows to have been exposed to infection from a notifiable disease … [including] smallpox, cholera, diphtheria, …” And so on. The list of diseases includes things against which we are now routinely vaccinated (diphtheria, tuberculosis), those we haven’t seen in the UK in my lifetime (typhus), and those most people would now brush-off with a course of antibiotics (flu, pneumonia). Except, of course, that at the time there was far less vaccination and no antibiotics at all, since mass production didn’t start until the 1940’s. So as well as being an observation of a period in history from a point in time from which its significance was only poorly understood, this book is a time capsule from a period when diseases were a lot bigger and broader threat than they are now.