Flash Boys

Michael Lewis (2014)

Another of Michael Lewis’ now-classic tales of Wall Street misadventure, this one focusing on the all-but-unseen - and even less understood - growth of high-frequency trading (HFT). This one follows the efforts of a small group of insiders to create a new stock exchange that’s immune both by policy and by design to the arbitrage and strategies HFT uses to game the conventional exchanges. A fascinating list of characters cross the pages, including Russian programmers, disaffected traders, and a network guy from Dublin - all working to make a system that was paying them well disappear in the interests of fairness (and their own long-term financial gain).

At one level this book is less satisfying than The Big Short: Inside the Doomsday Machine, perhaps because the story still hasn’t finished. The reader is left wanting to know the fate of the new IEX exchange, and the way the market changed as a result. For a techie, it’s also unsatisfying that so much of the technology remains unexplored, although it would obviously have made the book inaccessible to anyone but a computer junkie: perhaps there’s a much more technical follow-up that could be written.

Although the story mainly revolves a case of market failure - high-frequency traders capturing huge value while taking no risk and providing no real advantage - it’s also in a strange way an example of market success, when Goldman Sachs and other banks realise that their support of HFT is simply too risky for the gains they’re capturing themselves. There’s also an irony in the banks’ worrying that, in the case of another crash, the banks will take the losses while the HFT firms walk away with the gains - which is exactly the reverse of the situation after the 2008 crash, where the public took up the banks’ bad debts. Whether this is a sign of things to come is hard to decide, but it does show how even the most dysfunctional system can be changed when people recognise its dysfunction and are prepared to act to remediate it.

4/5. Finished Saturday 3 May, 2014.

(Originally published on Goodreads.)

Let’s teach everyone about big data

Demolishing the straw men of big data. This post comes about from reading Tim Harford’s opinion piece in the Financial Times in which he offers a critique of “big data”, the idea that we can perform all the science we want to simply by collecting large datasets and then letting machine learning and other algorithms loose on it. Harford deploys a whole range of criticisms against this claim, all of which are perfectly valid: sampling bias will render a lot of datasets worthless; correlations will appear without causation; the search goes on without hypotheses to guide it, and so isn’t well-founded in falsifiable predictions; and an investigator without a solid background in the science underlying the data is going to have no way to correct these errors. The critique is, in other words, damning. The only problem is, that’s not what most scientists with an interest in data-intensive research are claiming to do. Let’s consider the biggest data-driven project to date, the Large Hadron Collider’s search for the Higgs boson. This project involved building a huge experiment that then generated huge data volumes that were trawled for the signature of Higgs interactions. The challenge was so great that the consortium had to develop new computer architectures, data storage, and triage techniques just to keep up with the avalanche of data. None of this was, however, an “hypothesis-free” search through the data for correlation. On the contrary, the theory underlying the search for the Higgs made quite definite predictions as to what its signature should look like. Nonetheless, there would have been no way of confirming or refuting the correctness of those predictions without collecting the data volumes necessary to make the signal stand out from the noise. That’s data-intensive research: using new data-driven techniques to confirm or refute hypotheses about the world. It gives us another suite of techniques to deploy, changing both the way we do science and the science that we do. It doesn’t replace the other ways of doing science, any more than the introduction of any other technology necessarily invalidates hat came before. Microscopes did not remove the need for, or value of, searching for or classifying new species: they just provided a new, complementary approach to both. That’s not to say that all the big data propositions are equally appropriate, and I’m certainly with Harford in the view that approaches like Google Flu are deeply and fundamentally flawed, over-hyped attempts to grab the limelight. Where he and I diverge is that Harford is worried that all data-driven research falls into this category, and that’s clearly not true. He may be right that a lot of big data research is a corporate plot to re-direct science, but he’s wrong to worry that all projects working with big data are similarly driven. I’ve argued before that “data scientist” is a nonsense term, and I still think so. Data-driven research is just research, and needs the same skills of understanding and critical thinking. The fact that some companies and others with agendas are hijacking the term worries me a little, but in reality is no more significant than the New Age movement’s hijacking of terms like “energy” and “quantum” — and one doesn’t stop doing physics because of that. In fact, I think Harford’s critique is a valuable and significant contribution to the debate precisely because it highlights the need for understanding beyond the data: it’s essentially a call for scientists to only use data-driven techniques in the service of science, not as a replacement for it. An argument, in other words, for a broadly-based education in data-driven techniques for all scientists, and indeed all researchers, since the techniques are equally (if not more) applicable to social sciences and humanities. The new techniques open-up new areas, and we have to understand their strengths and limitations, and use them to bring our subjects forwards — not simply step away because we’re afraid of their potential for misuse. UPDATE 7Apr2014: An opinion piece in the New York Times agrees: “big data can work well as an adjunct to scientific inquiry but rarely succeeds as a wholesale replacement.” The number of statistical land mines is enormous, but the right approach is to be aware of them and make the general research community aware too, so we can use the data properly and to best effect.

The Psychopath Test: A Journey Through the Madness Industry

Jon Ronson (2011)

3/5. Finished Wednesday 2 April, 2014.

(Originally published on Goodreads.)

I Spend Therefore I Am: How Economics Has Changed the Way We Think and Feel

Philip Roscoe (2014)

A fantastic exploration of the purpose and dangers of economics and its effects on society.

(Full disclosure: like Philip Roscoe, I work at the University of St Andrews, and we know each other slightly.)

Roscoe’s main observation is of a fallacy, that economics is a descriptive endeavour. Instead, he argues that it becomes normative: it causes people to enact the behaviour that it thinks it simply describes. Introducing an economic model into a situation causes those concerned to act in accordance with the model, rather than (as we often think) the model simply exposing behaviours and motivations that were there already. Roascoe supports this hypothesis with a fabulous range of examples, from Norwegian fishermen to prostitution and internet dating.

In many ways this argument is reminiscent of that made by Adam Curtis in The Trap. Mathematical and scientific models are necessarily abstractions of the agents and phenomena they describe, but when these models are used to design systems, we can find that – rather than the systems failing because the models are incomplete – the people involved respond to the systems’ incentives by becoming more like the simplified abstractions. What game theory is for Curtis, economics is for Roscoe: simplifying assumptions lead to a reduction in human richness and behaviour through the power of incentives.

One can read this book as a call to arms to reclaim economics (and society) by embracing these characteristics about systems and people, and to design systems with this power in mind: systems that enrich and encourage people rather than simply following the dictates of money and its tendency to further enrich the already wealthy at the expense of the poor. That would be a noble outcome for what is a fascinating piece of work

5/5. Finished Saturday 29 March, 2014.

(Originally published on Goodreads.)

Writing on the Wall: Social Media - The First 2,000 Years

Tom Standage (2013)

A tour through the history of media, beginning with Cicero’s speeches and letters and continuing through to the familiar Facebook and Twitter.

The author asks a simple question: are social media, and the questions they raise about people’s interaction styles, really new phenomena? He asked similar questions about the internet as a whole in The Victorian Internet, his (excellent) history of the telegraph system, and the answer he arrive at here is similar. Unlikely though it may seem at first glance, the majority of the history of media has actually been of social media, in the sense of information flowing informally along the relationships between individuals. The recent prominence of newspapers, radio, and television have blinded us to the fact that these broadcast media are actually historical anomalies, expressing a centralising and one-way tendency that is singularly unusual in the annals of human communication.

For me, the most interesting observation came in the epilogue. Through fretting about the impact of social media, families sometimes adopt strategies like “Unplugged Sunday” where internet-connected devices are banned in favour of more communal pursuits – and these pursuits may involve watching television together, making use of a technology that was previously targeted as the destroyer of family time. (The same might be said of reading novels, targeted in their turn in an earlier age.) Yesterday’s radical, disruptive technology becomes today’s comfort blanket sanctified by time and familiarity with startling regularity.

4/5. Finished Saturday 22 March, 2014.

(Originally published on Goodreads.)