Follow the reluctant adventures in the life of a Welsh astrophysicist sent around the world for some reason, wherein I photograph potatoes and destroy galaxies in the name of science. And don't forget about my website,

Saturday 23 March 2019

Review : The Square and the Tower

Niall Ferguson is certainly an interesting fellow. His books always follow a similar trajectory : for the most part carefully controlled, balanced and objective, but suddenly veering off into some weirdly right-wing tomfoolery at the end. One gets a sense that he's only able to maintain a certain level of self discipline for a finite time before he simply has to go on a rant or his gonads will explode, or something. That said, if you're a lefty like me and want some genuinely intelligent commentary from the right, you could do far worse than Niall Ferguson. And if you're the sort who believes that individual agency is paramount, this is a book you should read without delay.

This book is even more Fergusony than most. The influence of society and its networks on how we think and act has been a growing theme in this blog of late, culminating in this monster of a post (with a shorter version now available here, albeit still subject to tweaking). I've been interested in networks as a way of understanding information flow, but Ferguson takes a very much broader scope - everything from economics to disease transmission.

In The Square and the Tower, Ferguson has it that society is dominated by two main modes of networks : hierarchical towers, in which information flows down from above, and market squares, in which things are more egalitarian and everything less dependent on centralised control. It's a very nice analogy indeed.

Is it any good ?

Throughout most of the book, Ferguson is mostly in careful control of what he writes regarding the pros and cons of each system. He's generally against hierarchies and more in favour of "free market"-style square approaches, though he notes, for example, that WWII could not have been won by a network. Yet at the end this all collapses into a mess of contradictory conclusions, especially when it comes to regulating social media (which he blames for just about all the ills of the modern world). One gets the impression that it was written in haste and probably could have used a co-author - it's very clear he's a historian, not a sociologist.

On the positive side, like all his books it's extremely readable and eloquent, without dumbing down. Sometimes I thought it might be nice to presume a little less knowledge on the reader's part, as he has this tendency to start commenting on random bits of history as though everyone were already aware of the basics. His goal of examining how networks and hierarchies have different effects and vulnerabilities is a laudable one, and for the most part he picks out good examples in support of his case. Where he makes a good point, he usually does it very well.

The main problem of the book is that he doesn't do this nearly enough. He frequently gets distracted by various anecdotal incidents, losing track of the main thrust of the text and going off on unrelated tangents. This isn't as bad as it could be because these tangents are usually interesting (and readable) in their own right, but they don't add anything obvious. If there's a point to these, he needs to state it directly, but doesn't. The whole thing ends up feeling like a very solid prelude, but a missed opportunity to write a true masterpiece : something that would lay out all the key findings of sociology about how different types of network operate, how the structural shape of society determines how it operates and influences the beliefs and goals of its participants. It's a good book, no doubt about that, but it misses the goal - or at least what I saw as the obvious goal - by a long shot. I give it 7/10 overall.

What does the book actually say ?

It took me a little while to find a decent review of the book's contents to save me the trouble of having to write a lengthy post myself. For some reason, several other reviewers seem terribly confused by the opening description of the Illuminati. This is pretty obviously just an attempt at illustrating the pros and cons of a network : how it could rapidly succeed in some circumstances but also remain extremely vulnerable. On the other hand others are full of gushing praise which I thought was just weird as the flaws of the book are not all that subtle.

Fortunately there are two good reviews and summaries. First, there's a presentation (slides only) by Ferguson himself, which does a much better job of of summarising the general conclusions than he did in the book. Second there's this review article by a sociologist, which offers a critical but fair appraisal giving credit where credit is due and correcting mistakes where present.

Ferguson says in his presentation that he learned six major things while writing the book :

1.) Birds of a feather flock together
Ferguson notes that like-minded people tend to associate with each other, which naturally leads to filter bubbles and polarisation. On the other hand he notes that this can be limited : people can hang out because they share specific common interests, not necessarily because they're alike in every way. They also have very different connections to each other, e.g. familial, economic, e-mail, etc.

2.) There's strength in weak ties - leaders don't always matter
Few men are islands, though some have fewer connections than others. We might expect the leaders of major organisations to be the most influential because they have the most connections to the most people, but it's more subtle than that. Leaders can sometimes have dense connections within their own groups but few connections to other organisations. In contrast, some group members can be connected to multiple groups. These connections which let networks interact can be crucial, even if they're not particularly strong.

Ferguson gives the example of the American War of Independence. Paul Revere's message could not have spread so quickly, he argues, if he wasn't as connected to different groups. Likewise while utterly dismissing conspiracy theories about the Freemasons controlling all of history, Ferguson points out that they did have an important role at one point - their numbers weren't great, but they didn't need to be since their connections were spread into multiple important social groups.

3.) Structure defines how information spreads
I've covered this one before in detail, but the basics of it go like this. In a hierarchical structure, ideas can spread more virally because there are fewer connections. Most ideas do not have universal appeal and how we evaluate their validity and interest depends on how are friends react. The greater the fraction of our contacts who believe an idea, the greater the credibility we give it. Members of a hierarchy have far fewer friends than in a complex network, so information spreads much more rapidly (if you only know two people and one of them believes that chickens are evil, that's a full 50% of your sources telling you that chickens are evil - whereas if it's one out of fifty, that's a negligible fraction of your sources telling you the same weird thing). Being more social does not automatically make information spread more quickly.

Ferguson has this backwards, saying that ideas spread less rapidly in hierarchies where information can only flow vertically. I would say that this rather hints that the spread of an idea is a complex function both of the idea itself and the network it's embedded in. An idea that a hierarchy approves of can spread much more rapidly because lines of communication are much more direct. On the other hand if one agent has an idea that does not meet with approval, it's much easier for any one level of the system to stop it spreading any further. Conversely in a network, ideas which have truly universal appeal can spread fantastically quickly, whereas those that don't get diluted by their spread and are quickly stifled. Were this not the case, my website would have a lot more hits. :P

4.) Networks are not static
How rapidly they change depends in part on the structure of the network. Some, such as the early Protestants, are highly robust to the loss of large numbers of connections : the basic shape remains unchanged. Others can be highly variable, with adding just a few connections able to change the shape completely. Martyrdom of a prominent individual may or may not attract supporters - that is not inevitable, but a function of the network shape.

Some networks can be highly vulnerable to hierarchical suppression or have such strong interdependency that a single failure can collapse the whole thing : states that appear stable, such as the Soviet Union, can actually have fatal weaknesses that make them vulnerable to a domino effect. Interdependency poses something of a paradox, bringing its participants closer together and stronger to some external threats, but curiously vulnerable to others, as illustrated by the financial crisis of 2008. Ferguson notes that this was exacerbated by the financial structures tending to amplify fluctuations : an emergent property of the network itself and not due to any one node.

Yet other networks are much more robust, and in those cases...

5.) It takes a network to beat a network
Ferguson gives the example of British general Walter Colyear Walker as a case of fighting fire with fire. Rather than attempting to fight jungle guerrillas with conventional tactics, Walker realised he needed to "own the jungle". Killing a few soldiers wouldn't collapse the enemy network. Instead, Walker strove to win local hearts and minds as much as possible, relying on local intelligence to locate the enemy. He also had his troops become, in effect, guerrillas themselves, with a decentralised command structure that enabled flexible, on-the-spot decision making without needing to run everything by superiors first.

This is more than simply beating the enemy at their own game. It's also about recognising why the enemy network works so well : every agent benefits from but is not dependent on every other agent. Every single guerrilla is committed to the cause and the network is self-organising, so killing one or two doesn't diminish the overall fighting capacity as taking out the general of a conventional army would.

6.) Networks can promote more extreme inequality than hierarchies
Celebrities who are famous for being famous are a variant of the Matthew Effect :
"For unto every one that hath shall be given, and he shall have abundance: but from him that hath not shall be taken away even that which he hath."
Connections breed connections through metaknowledge. Popularity is taken as a sign of credibility. Awards beget more awards because all further work is seen through the lens of enhanced prestige. Wealth makes it easier to amass yet more wealth. Ferguson is particularly keen to emphasise the problems of this last one with regard to the tech giants of today - not without justification. He notes that market forces often disrupt traditional hereditary networks, allowing for meritocratic social mobility. The reason this hasn't happened here is because networks and market forces are aligned, promoting an explosion in wealth inequality neither desired nor foreseen by its generally young and left-leaning users. It's certainly interesting to hear a clearly right-wing commentator attacking the free market and wealth inequality... I have the distinct impression that writing this book must have left him tremendously confused.

From the slideshow. Ferguson is certainly not one of these free speech absolutist types that somehow still plague the internet. Yet this withering put-down is ill-deserved : if anyone was warning about the dangers of internet-based social networks before they became a thing, they were barely heard.

There are other points of inconsistency and confusion. The "no man is an island" principle is implicit throughout the book. This is rather at odds with Ferguson's right-wing tendencies, which are normally associated with a respect for personal freedom, responsibility and autonomy, whereas here it's clear that everyone is directly dependent on everyone else. Or is it ? Ferguson notes that hierarchies have been the norm throughout human history until relatively recently, but this is refuted in the review :
Ferguson is simply wrong to say that “for most of history, life has been hierarchical” and to claim that in primitive tribes someone needs to decide for the group when to perform tasks and how to allocate resources. This is basically true of agricultural city-states from Hammurabi’s Babylon to Kamehameha’s Hawaii, but you will be hard pressed to find an anthropologist who finds it consistent with the experiences of stateless hunter-gatherers. (The support Ferguson cites for this argument is a computational model from theoretical biology that makes unrealistic assumptions about the distribution of wealth in a primitive context.)
Similarly,  Francis Pryor notes in Britain B.C. that in early societies in Britain appear to be profoundly egalitarian, with the only distinguishing feature of the rulers being that they lived in a slightly larger roundhouse. Yet reading between the lines I felt there was a frequently unspoken question : to what extent are networks simply an inevitable result of their participants, more of a symptom than a cause of behaviour ? In primitive tribes, a network structure to society may be unavoidable due (perhaps) to the small number of agents and their mutual interdependency : society forms the network rather than the network forming society. This seemed apparent throughout many of the other examples Ferguson gives - that sometimes the network is more the result of actions than the cause of them. Often there could be interplay between the two, but I would have liked more discussion about when and why this happens.

Conclusion :  the future

Ferguson does a great job of explaining why networks act as they do, in particular by applying formal network analysis to historic examples (although I'm not sure why he's so obsessed with Henry Kissinger). While he provides many illustrations of those networks, which are sometimes helpful, they're often under-utilised and don't really add as much as they could. What was sorely lacking was much in the way of diagrams of the large-scale organisational structures. Looking at the fine details is interesting but makes it hard to get a sense of the overall properties, and while the descriptions are helpful, diagrams would have added a lot more.

Generally though, it's hard for me as a lay person to spot any serious deficiencies in the first three quarters or so of the book. It's in the final section where things get most provocative. I came away with the impression of having plucked an unripe (or possibly slightly decaying) fruit - there was a definite sense of, "you've definitely got something here... something that's potentially extremely important, but your arrow has fallen just short of its target." Which is not unreasonable when you're trying to determine what's gone wrong with the Western world and what's going to happen next.

One such example is his notion of the "administrative state", in which governments pander to demands for action by creating more bureaucracy. By creating ever more internal connections that accomplish little, they create the appearance but not the substance of useful activity. Such behaviour could clearly be both the consequence of some deeper underlying failure and the cause of further problems.

This certainly has a ring of truth about it, but lacks anything convincing as to why this development has occurred. "A fundamental deterioration of standards in both legislation and governance that we see in nearly every democracy... professional politicians are more concerned with spin than substance..." - this is a mere description (and a tautologous one at that), not an explanation. So why should democracies progress to this over time ? What is it about their network structure that drives this ?

Likewise, Ferguson makes a compelling case for the importance of social media in the current state of affairs, but he goes too far. As the review says :
Similarly, Ferguson describes Brexit and the Drumpf campaign as network-based. They were certainly insurgent campaigns opposed by political establishments, but this does not necessarily mean that they were network-based: Both campaigns seem to have benefited at least as much from skilful political entrepreneurship of a more conventional kind. Ferguson shows that Drumpf had a much more prominent social-media presence than Clinton and notes that Clinton outspent Drumpf considerably, but he ignores that television had a huge role in Drumpf’s success. The TV channels were eager to provide coverage of him, which the media-tracking firm mediaQuant estimated as equivalent to almost $5 billion in advertising. Again, that Drumpf had fascinating and well-attended rallies that television channels found worth broadcasting gave him a great advantage, but it didn’t really have to do with networks.
This echoes what I've said for a long time : that social media played a part, but that the role of the mainstream media is far greater. Specifically, while targeted persuasion can be remarkably powerful, you can't convince people of true absurdities (like the idea the Donald Trump is presidential material or that water isn't wet) without first indoctrinating them. That doesn't mean social media might not have been responsible for landing the killer blow (i.e. the swing states in the Trump election), but the major wounds were done by decades of hyperpartisan media bullshitting.

In Ferguson's view, that social media didn't much affect certain demographics, like older voters, doesn't matter because of secondary affects. How social media information gets off the net and spreads through the real world through more established channels is something I think urgently needs investigating; I have a hard time believing it could be the dominant factor at work in the world today but I could be persuaded.

However, where Ferguson does better - where I think he may be on the cusp of something very important indeed - is in noting that globalisation has been driven by technological developments, that there has been a huge growth of a network far more important than Facebook, that the tech giants are more a consequence of this than a cause : "it was precisely the relaxation of central control that made the American information technology revolution possible". So the shift away from a hierarchical governance begat the creation of even more networks, with complex feedback effects. I can't help but recall this well-known graph :

Could it be as simple as a change in economic policies that got us where we are today ? A shift from control to de-regulated networks that drove economic and technological growth, and all the mixed blessings that that brings ? I don't know, nor do I have any idea which specific economic policies might be responsible. It's an awfully tempting idea though.

So I will venture that what I think Ferguson is (somewhat reluctantly) getting at is this : sometime in the 1970s (for reasons unknown), governments started to relax their grip on the reins. International trade flourished and technological development prospered, in particular, the deliberate creation of the decentralised internet. That led to increasing inequality (Feguson has some very interesting caveats about popular claims that global inequality is actually falling) which is the primary cause of dissatisfaction in the world today. Workers are not being fairly rewarded for their efforts, and they know it, whilst simultaneously, the major new communications channels (but also in no small part the traditional media) have been more successful at finding scapegoats than extolling the benefits of the changes - that is what the news tends to focus on because there's a psychological preference for it.

That's my guess, based on Ferguson. I think it's a reasonable guess but I'd love to hear other (perhaps more informed) opinions.

If so, it follows that the main solution to the current problems is, in one sense, more economic regulation designed to ensure the increased wealth is more fairly distributed. That, in turn, eliminates the need to find scapegoats and the appeal of the ever-present xenophobia is rendered impotent. It is not that making people richer makes them nicer, it's that treating them fairly makes them more reasonable. This has to be done consistently, bearing in mind that sometimes fairness entails punishment.

I say, "in one sense" because firstly it's by no means clear what was driving this economic shift. Was it a simple, arbitrary policy choice driven by ideologies or pragmatism ? Or was some other, deeper network shift occurring to incite it ? Secondly, it remains unclear as to whether it's regulation of economics that will drive change of policies and actions, or if regulation of information that will drive changes of economics. Perhaps either would work equally well, or perhaps there's a fearful asymmetry to them.

I've offered up my own speculations here because Ferguson is either unwilling or unable to draw a coherent conclusion. He simultaneously decries the loss of hierarchical oversight on the technology giants and social media networks but lambasts governments for attempting to regulate social media content, saying, weirdly, that this amounts to giving corporations even more control over citizens. This makes little sense to me and feel more like Ferguson is wrestling between his own ideologies and the natural conclusions of network analysis.

While Ferguson has some interesting points towards the end, he mostly degenerates into pat, trite statements wailing about how everything is getting worse because people are just awful. This I find unhelpful. Much more worthy are the general conclusions and questions raised in (and by) the book overall : the extent to which networks are causal and consequential; how they can sometimes be vulnerable and sometimes highly robust; when hierarchies are successful and powerful and when networks triumph. Above all, the main lesson is surely that quantitative, objective network analysis can reveal unexpected connections and consequences. Perhaps, one day, we'll be able to use this to design a robust, stable network to help maintain a fair and prosperous society. For now a more urgent task wold be to examine if our network is one of those that can survive remarkable levels of disruption, or if it's the kind that appears stable but is actually incredibly vulnerable. Maybe all the current hoo-hah is just so much useless noise, maybe it's presaging a catastrophe - or maybe we're living through a phase transition from one state to another. Whether that change will ultimately be an improvement or not remains to be seen.

Sunday 17 March 2019

O Rhysy where art thou ?

Are you one of the tiny handful of people who only follow me on this blog ?

Have you been wondering where I've been and why posts are even fewer lately than they normally are ?

Have you been secretly dreading that I must be preparing some absolute monster of a post that's gonna need a well-trained team of six stout mountaineers and a harpoon just to even read the abstract ?

Worry no more ! The reason has simply been that Google are a bunch of bastards who decided to cancel one of their greatest products : Google Plus. I shall save a paean on that for another day. Suffice to say that with the final shutdown approaching rapidly (2nd April), I did not particularly want to just watch my thousands of posts vanish into oblivion. Instead, using the Google Plus Exporter tool, I've been saving my assorted ravings and archiving into them into not one but two shiny new blogs.

Physicists of the Caribbean has evolved steadily over the years from wanton silliness to scientific analysis to philosophical ramblings. Posts here have also tended to spiral out of control in terms of length. See, what Google Plus let me do, apart from reaching an audience of some quite wonderful people, was jot down quick initial impressions without feeling obliged to explore anything in excessive detail. I've found that to be an incredibly useful experience and I don't want to lose it. Nor do I want to start doing the same thing on other social media only to potentially lose it all again one day.

That's where the new blogs come in. These are not just static archives but living notebooks where I can and do scribble down something quickly for later use - many of the posts here on POTC have evolved from these kinds of snippets when I've sensed that some general trend can be seen from specific examples.

I'm also aware that different people are interested in different things and not everyone wants to subscribe to a blog where the content is so incredibly randomised. But having a different blog for each topic doesn't suit me either. So here's how things stand :

  • First and foremost is this one : a place to write finished pieces, but also now with an increased effort on concision and, I hope, more silliness again.
  • Second, there's Little Physicists. That one focuses heavily on astronomy and data visualisation - stuff I can comment on professionally and quick, rough-and-ready data visualisation tests. 
  • Thirdly, the most active blog is Decoherency, which is basically a shorter version of this blog : everything else I'm interested in, but with much shorter and more focused posts. Lots of things about the latest political developments, rational thinking, nature, and whatever else I happen to find interesting.

These two new blogs are already active, so check them out, follow them if you like, and reshare anything you find interesting because it sure as hell won't reshare itself. Both still have "placeholder" posts which need updating and I'll also be going through all three blogs to replace links to Google Plus posts that might soon disappear. It's been a truly enormous amount of work to archive years of material, but it's now basically complete. And if you want to follow me directly on social media, you can find a complete list of my internet locations here.