Uncompromising Men

Or : How rational thinking won't stop you from being stupid

Uncompromising men are easy to admire. He has courage; so does a dog. But it is exactly the ability to compromise that makes a man noble.

It's unfortunate that this line is found in a) a Mel Gibson film and b) is said by a villain, because it's true. People naturally prefer clarity to uncertainty. We like to divide the world into black and white, right and wrong. This is the truth, that is just stupid. If you believe that, you must be an idiot.

That of course is why Braveheart is a popular film - and the same goes for Star Wars, The Lord of the Rings, etc. The goodies are goodies and the baddies are baddies, with all the moral ambiguity of a bowl of custard. And even though there are plenty of very successful stories with much greater moral complexity, the simplistic version never really loses its appeal.

But in the real world anyone with an ounce of sense knows, deep down, that it's much more complicated than that. Granny isn't an evil person because she's a little bit racist*. The high school physics teacher isn't certifiable because he sympathises with Moon landing conspiracy theorists*. And the local shopkeeper isn't a nutter because he thinks global warming is all some kind of hoax to raise taxes. There's a world of difference between holding an idiotic (or simply mistaken) belief and being an idiot.

* Personal examples. I also had a teacher who thought that things float in space because of "thin air" and another who thought that the stars shone by the reflected light of the Sun.

Deep down, we all know this to be true. But sometimes things get confusing.


Being Rational Is Not Enough

We all have an innate sense of curiosity, but critical thinking demands more than a simple desire to find things out. We have to constantly watch for our own biases, to check we're not believing something just because we want it to be true. It's not an easy skill, and unlike history or French or line dancing, it's not really something you can teach in a specific lesson. On the other hand, statistical thinking doesn't come naturally at all, but that definitely is something you can and should teach.

We tend to learn by induction - we see (or are told of) patterns and trends, and assume the're generally or always true in new situations. The classic example is that if we see a hundred white swans, we assume that all swans are white. Generally this is pretty sensible - after all, it's not practical for us to check every single swan on the planet - provided that if we do find a black swan we correct our opinion.

The difficulty with thinking statistically is that our own personal experiences are always true. When someone says something that's in flat contradiction to what we've seen with our own eyes, there is a certain logic in denying it. "You can't possibly have seen a black swan, you must be a very stupid person."The problem is that that's equally true from their perspective, but remembering this isn't easy - instinct takes over. It's even more difficult to realise that your experiences will have been influenced by a thousand different factors, and there's no reason to suppose that those will be the same everywhere.

Thinking in this way demands constant vigilance. Sometimes we can make those little personal exceptions : Granny might not be too fond of black people, but she's basically OK... that one swan was pink, yes, but - oh, wait, that was a flamingo. Whoops.

I AM NOT A SWAN.

Statistical Thinking Needs To Be Taught

When we find an anomaly, we don't automatically say, "All my previous observations were unusual, this new thing is normal." Instead we assume that it's the new thing that's the anomaly. If you don't see any birds other than swans your whole life, then one day when you're six your parents take you away from the swannery where you grew up to visit the zoo and you see a flamingo, you aren't going to say, "wow a new type of bird !" You're going to say, "What a funny-looking swan !" And then your parents are going to feel like right idiots for apparently forgetting to teach you anything much at all, and you may be well on your way to becoming a bit of a thickie.

Thinking about what sort of selection effects are at work does not come naturally, and it's difficult. Because we tend to say that anything unusual to us is unusual overall (rather than assuming we've been in an unusual position), things that don't fit the pattern don't necessarily challenge our ideologies. If we find more examples, we just say there are more exceptions, more mitigating factors. The whole thing becomes unwieldy and complex. We don't like it. It was all so much simpler before the "anomalies" cropped up. We know there are exceptions, but we don't even really believe those are evidence that our assumptions are wrong because there are special exemptions.


Simple Messages... For Simple People ?

Then along comes a voice full of sound and fury that cuts through all that like a scalpel. Should we tighten border security to prevent foreign terrorists from entering ? NO LET'S JUST SEAL THE BORDERS COMPLETELY ! In fact, let's drive the English out of Scootland forever ! Ach ! FREEEEDOM !!!!

Of course it bloody didn't. The UK is a democratic country, and has been for quite a while. Scotland is not under the rule of a tyrannical English king and hasn't been for centuries. It was monstrously, disingenuously simple to claim that everything would have just magically have gotten better by breaking the union that's worked pretty well* for the last 300 years. But damn it was an appealing message, if you didn't stop to think about it for five minutes.

* It conquered the largest empire the world has ever seen and managed to lose it without descending into anarchy, and I call that a success.

Complicated messages are difficult to sell. "This new tax policy slightly favours the rich, except for those who don't employ more than seventeen butlers and only if they've never been to Zimbabwe with a goose" is just not as appealing as, "THIS POLICY HURTS THE POOR !" But the reality is that things are rarely so simple - their are usually caveats to everything.

Simple messages have the virtue of getting to the heart of the matter, to what we're really thinking beneath all those layers of exceptions we've accumulated. It's especially potent when it shifts the blame to someone else. So, "it's because of constant Western support for dictators due to profits from selling them weapons" becomes, "it's because dey is Musilms, innit !"


Being Rational Won't Save You From Stupidity

Now, to anyone thinking statistically, the second line of reasoning is laughably absurd. Simply put, if being a Muslim were the cause of violent extremism, we'd all be dead by now. But that point of view is understandable if all you see in the media is jihadist after jihadist, as opposed to, say, stories like this or this. It's as natural a conclusion to draw as that all swans are white. I've come to conclusions in a similar way myself. My point is that anyone blaming Muslims may have an idiotic point of view, but that doesn't automatically make them an idiot. They're coming to their conclusion by a process of reasoning which is actually perfectly rational, but woefully incomplete.

When confronted with the the reality that actually things are a bit more complicated than that, there are essentially two ways to proceed. The first is to say, "Oh, sorry, I was wrong." The second is to say, "GET LOST YOU COMMIE BASTARD I MEANT EVERY WORD OF IT !".

Now, in a rational world, one would think that the first option would be more admirable. Being prepared to admit your error and face the wrath of your enemies for doing so should be seen as courageous, but when did you ever see the headline, "POLITICIAN GUILTY OF LEARNING !" ?You never did, partially because there's a fundamental misunderstanding that intelligence means knowing all the right answers. Perhaps that's even true to some degree when you're ten and you're marked in school questions as right or wrong. It is not true in the grown-up world where there may not even be an answer at all, let alone one that's right or wrong.

This, I take it, gentlemen, is the degree, and this the nature of my advantage over the rest of mankind, and if I were to claim to be wiser than my neighbour in any respect, it would be in this - that not possessing any real knowledge of what comes after death, I am also conscious that I do not possess it.

Being aware that you don't know the answer doesn't make you stupid... unless there actually is an answer and you refuse to accept it : "NO MUMMY THERE'S NO SUCH THING AS FLAMINGOS !". Actually, when an issue is genuinely not yet settled, being certain of the answer is a sign of stupidity. While ideally you shouldn't be certain of uncertain things to begin with, it's still far better to change your opinion when confronted with good evidence - even if it takes you a long time to do so.


The Virtue Of Being Wrong

This is something we leave it until far too late to teach people : that research only happens because we don't know the answer, that while we might have simple answers to many children's questions (no, the Earth is not flat), when it comes to cutting-edge research things are altogether different. We forget that we can't learn new things if we already know the truth - we can't better ourselves without first admitting our flaws. It is, after all, somewhat comforting to believe we're already perfect.

That's only part of it. Admitting our mistakes is seen as a weakness, but on the other end, standing by our beliefs is seen as a sign of strength. And it does take courage to say unpopular, perhaps unpleasant things, or at least things which will cause strong reactions. But it appears to me that while we admire the courage of convictions, we almost never acknowledge the courage to change those convictions - which would probably go a long way to making the world a nicer place.

Sticking to your guns despite evidence to the contrary might be a brave thing to do, but it isn't rational. Nevertheless, a skilled, charismatic individual can use this perceived bravery to deadly advantage. The outcries from their opponents are deemed to reveal their own irrational, biased views : those "bleeding heart" liberals again ! See how they hate our righteous cause, they must be desperate to say such stupid things ! They're bound to disagree with us !


Manipulation

Taken all elements together - a simple but rational (though incomplete) and blame-shifting message, the lack of respect for people who change their mind, the perceived courage of maintaining one's ground in the face of attack, and the ability to make the oppositions's attacks look like a sign of bias, and disagreement become basically impossible. The attacks become stronger and stronger signs that the opposition are acting out of hate, not reason. The more outrageous the statements and the greater the refusal to back down, the more attacks are made, and the worse anyone disagreeing looks. Uncompromising men are not only easy to admire, they can be difficult to stop admiring.

A cult of ignorance probably doesn't look like that to its followers. It probably looks like someone is finally being honest enough to say what they're really thinking (despite the fact that that is not a virtue), to have the courage to agree with their entirely rational but uninformed conclusions, to circumvent the apparently highly contrived excuses of an intellectual elite. It does not necessarily happen simply because people are stupid.

It should be obvious by now what the end result of all this is.

Trump himself has no ideologies at all as far as I can tell, but he inspires those ideologies in others.

How do we break this madness ? I'm not sure, though I offer some suggestions as to what works to change my mind at the end of this post. It's important to remember that while people aren't entirely rational creatures, they're not entirely irrational either. I suggest not to play Trump's game. Don't leap in screaming how Trump is a moron, it only makes you look biased. Instead, chip away at his arguments one by one, arguing first towards doubt, not the extreme opposing viewpoint.

What I'm more sure about is how we can stop such utter lunatics rising to prominence in the future. Tony Blair, for all his many faults, had the right of it : education, education, education.

• Statistics, statistics, statistics. We need this to be taught in primary schools - not the maths, but the methods : big-picture thinking, selection effects, etc. Statistics is also a good way to teach important aspects of critical thinking.
• Better humanities classes in schools. Two things I strongly recommend : analysing adverts to see how people are manipulating you, and debates in which everyone is forced to advocate positions against their own viewpoint. Look at controversial, hard-hitting issues from a young age.
• Compulsory study of Plato's Apology in early high school, maybe selected passages in primary school. There aren't many examples of martyrs to the right to question rather than the right to know, but this is one and it's absolutely magnificent.
• Make children aware that what they're learning is a simplification. For god's sake tell them that the Solar System model of the atom isn't the final answer, and more importantly, that we don't have a final answer on many questions, that certainty is seldom justified outside the classroom.
• Break up news corporations. No-one should have a monopoly on the truth. No individual should be able to control more than a single newspaper or television channel.
• Continuously remind children never to be afraid of ugly facts... but at the same time, leaping to conclusions because of a single fact is not usually a good idea.
• Make everyone watch more Doctor Who.

The Value Of Nothing

Or, What's The Point Of Knowing Useless Things ?

It's not the greatest movie ever made, but there is at least one valuable scene in Kingdom of Heaven :

We might, perhaps, say the same about knowledge. Not knowledge of any particular topic, but knowledge in general - and more than that, understanding and wisdom.

There are some who claim a monopoly on truth - that their theory trumps that of mainstream science and that conventional research is nothing but a waste. They utterly fail to understand that progress without disagreement, without permitting and encouraging the exploration of the other point of view, is impossible. Worse, they insist that anyone who disagrees with them must be a closed-minded fool, and fail to appreciate the massive irony of the situation since their own theory has produced no tangible consequences whatsoever. That their precious idea might be dismissed by so many people because it is simply wrong never enters their darkest dreams.

I have dealt with such notions at great length here, here, here, here, here and also here. This post is a bit different. I'm not going to look at why people are wrong to assume science is dogmatic. Instead I'm going to look at a particular and very widespread type of dogmatic thinking and its consequences.

There are those who believe that we should concentrate only on researching specific topics : tackling climate change, curing cancer, superconductivity... things with obvious, immediate and above all practical benefits. Who cares about some distant galaxy ? What possible use is knowledge of the mating habits of flamingos ? One of my grandmothers couldn't see the point in spending so much money on the Cassini mission. Even my own mother, who in most other respects I will proudly defend as one of the wisest individuals you could ever hope to meet, doesn't think we should bother with space colonization "until we've solved the problems on this planet".

Well mum, I hope you're reading.


Spin-offs

Technologies which do have direct social benefits are by far the easiest way of selling the value of pure, blue-skies research. Unexpected developments certainly do happen, and the consequences can be literally unimaginable until they actually occur. You're reading this thanks to the internet, partially developed during research into particle physics. Twenty years ago the net was mediocre at best. Today, it's probably impossible to calculate its contribution to the global economy, let alone the societal changes it's brought about. Think about that next time you want to say, "this research can't possibly produce anything useful". You're not psychic, you twerp.

Sometimes, science does not so much improve the economy (though it certainly does do that) as it does transform it out of all recognition. Can you imagine the world without modern medicine, electricity, fertilizers, disinfectants, mass transit, or telecommunications ? It would be, in that famous phrase, nasty brutish and short.

And yet in part because people fail to think statistically and take the long view, there is a belief that "chemicals" are bad for you, that anything "natural" is more beneficial... a damn stupid idea that takes all of three seconds to disprove.

Snake therapy : because acupuncture is too mainstream.

But I digress. Sorry. Anyway, while it's true that not all major technological breakthroughs have occurred as spin-offs from pure research, many have. Many that have occurred through more deliberate projects have still relied on theory developed through pure research. Even the value of spin-off discoveries from scientific endeavours is impossible to calculate - yes, even from the space program we should apparently ignore until we've solved everything down here.

It is true that for poor counties, developing a space program should hardly be their first priority - just as it might not have been sensible for them to have developed programs of world exploration back in the Middle Ages. But for those who can afford it, the riches up for grabs are almost literally beyond imagining.

Leaving aside the highly dubious issue of whether we can or will ever solve all the social problems here on Earth, space exploration is a means to improving life here on Earth - not an excuse to avoid its difficulties.


No, don't do that research ! Do this research instead !

OK, maybe the method of spin-offs isn't the most efficient method possible of making those practical technologies. This is very hard to prove. We already have - indeed have always had - a mixture of public and private research, sometimes focused on practical goals and sometimes on blue skies projects. But because of the interplay between the theoretical development (which is largely the province of pure research) and the practical development (which is often done by companies) it's extremely hard to judge if things could be made more efficient or not.

The problem with the idea that we should only focus on  "important, practical" research is that I have neither the interest nor the ability to solve medical problems. Non-scientists sometimes seem to think that "science" is some generic discipline, and that if you can do one sort of science you can do another. This is like saying that if you can row a boat, you can drive a Formula One racing car - or that you'd want to drive a Formula One racing car.

Medical problems bore me. Chemical reactions are incredibly tedious. Atmospheric physics leaves me cold. And to be honest, there are even huge sections of astronomy that leave me yawning. I'm just not interested in them, and I'm not going to become interested in them just because someone else thinks they're important. They are important, but that doesn't mean I want to research them any more than it means I want to become a pilot, an ambulance driver, a fireman or a policeman or any of the other very important professions available. We all have the right to pursue happiness.

(And while we're at it, how often do you see astronomers saying, "You mid-level accountants should stop being accountants and concentrate on fighting cancer ?")

So if all you're interested in is the economics, you can stop reading. Science's contribution to the economy cannot be overestimated. Prioritising specific areas of research is an incredibly arrogant approach, because you cannot possibly know which area will generate an unpredictable spin-off or discovery that might be important elsewhere. And sometimes those spin-offs, like the internet, are quite literally invaluable.

But while the technological benefits of science are innumerable, there are other reasons for pure research which are harder to sell but, perhaps, even more important. The great Arabic philosopher Al-Biruni summarised it thus :

"The stubborn critic would say : 'What is the benefit of these sciences ?' He does not know the virtue that distinguishes mankind from all the animals : it is knowledge, in general, which is pursued solely by man, and which is pursued for the sake of knowledge itself, because its acquisition is truly delightful, and is unlike the pleasures desirable from other pursuits. For the good cannot be brought forth, and evil cannot be avoided, except by knowledge. What benefit then is more vivid ? What use is more abundant ?”

Humility

If there's one inescapable lesson from astronomy, it's that we are small. No, not small. Pathetic.

Or rather, astronomy should be a humbling and character-building

experience. I, for one, can't stand character-building experiences.

Not just in space, but in time. Not only is our pale blue dot nothing more than a speck of dust, but we've inhabited it for only the briefest moment. To think - or rather, to be certain - that we are the pinnacle of creation or that the Universe was created especially and only for us is an act of monumental, breathtaking arrogance. To the Universe, were are nothing.

And yet while our total insignificance in the face of this vast, cool and unsympathetic cosmos demands our humility, and probably also means we shouldn't take life too seriously, it doesn't mean we have to cow down in fear or be stupefied into inaction.

So what's the value of astronomy ? Nothing ! Everything. If you don't think humility is an important lesson, then I can't help you.

So should we stop there ? We already know the Universe is unimaginably big and incomprehensibly old, what's the point of studying the boring details ? Well, we also know that most of the Universe is (most likely) made of a material about which we know almost nothing, we know an unknown force is driving its expansion, there are places in the Universe where time stands still, where mountains can't grow a millimetre high because the gravity is so strong, planets where it rains sulphuric acid and moons that tear themselves inside-out, we suspect that ours is not the only Universe and there might, just might, be connections between other realities, and we have absolutely no clue if we're alone in all this or if there are hordes of other intelligences waiting to be discovered. So, should be stop there ? Or shall we keep going, and find out how deep the rabbit hole goes ?

All we are is a kilo or so of warm blood-soaked goop sitting inside a skull. It is entirely possible that the true nature of the Universe is forever beyond our comprehension. But if we'd given up trying because the magnitude of the problem seemed too vast, we'd still be swinging through the trees.

To slightly modify an earlier quote of mine :

The thing is though, the real universe is full of giraffes, exploding stars, Scarlet Johannson, worlds covered in methane, pandas, stars so dense they slow down time, and cabbage. We can only survive in a minuscule region of space barely five miles thick, on top of a rock hurtling around an almost 100,000 mile-wide ball of plasma, and we think this is normal. Anyone who thinks astronomy is nothing but escapism should have their head shoved into a telescope until they realise just how dreadfully, pitiably small the so-called "real world" is.

Although teaching everyone to be humble, to step back and remember that they're just another stupid ape, is something an awful lot of people would do well to remember, there is perhaps another aspect of knowledge which is even more important. To really understand the value of knowledge, you must first be aware of the price of ignorance. But there is one final pragmatic concern I need to address first.


Won't someone please think of the children !

Specifically, the starving children. How, one might ask, can we afford to spend money on space probes when there are people dying of starvation in the world ? Is our "humility" really worth that much ? It's a legitimate question. I've already hinted at the answer, but just in case it isn't clear, let me spell it out more clearly.

First, research has produced the farming techniques which are so essential to modern life : fertilizers, pesticides, harvesting technologies, etc. Second, food production has been growing over time, and yet there are still starving people in the world. That alone proves that it isn't about prioritizing the wrong research or that money is being spent on the wrong areas - the problem of starvation is political. Science is responsible for generating greater food production, better medical care, and longer lifespans - bringing those benefits to everyone is the mandate of politicians, not that of researchers. Scientists are in no more a position to affect political change than anyone else.

And again, how many astronomers do you see saying, "You mid-level accountants are just a waste of resources and we should give all your money to starving children" ?

Thirdly, the specific charge that money is being "wasted" on frivolous research like geological and space exploration. Yes, in principle, you could stop spending money on rockets and spend it on food instead. But it makes no sense to pick on astronomers or geologists. Neither are particularly well-paid (except possibly privately-funded geologists who discover oil or valuable minerals). And the cost of the space program, even in America, is utterly dwarfed by other costs :

America spends less than $20 billion on NASA annually, but over $600 billion on defence. What's more sensible - scrapping NASA completely, or "slashing" the defence budget by 3% ? Privately, Americans spend more on tobacco, alcohol, illegal drugs, treatment for all those drugs, gambling, and even pizza than they do on space - and this is by far the largest space program in the world. Anyone who's genuinely concerned to solve social problems would do far better to cut down on junk food, booze and drugs than scrap the technology-generating space program. That, at a stroke, would simultaneously prevent a lot of health problems and free up money for the needy.

So pure research with little or no obvious practical benefits costs very little, doesn't prevent us from solving more urgent problems, and in fact offers solutions to such problems provided we have the wit to use its unintended breakthroughs correctly. And it teaches us humility, cautions us against arrogance, and warns us that our most deeply-held views could be overturned at any moment. Perhaps most importantly of all, we should remember what happens in a world without such free-thinking inquiry.


The Demon-Haunted World

There are some mistaken beliefs about the medieval world that refuse to die - they never believed the Earth was flat, for example. They did think it was the centre of the Universe, but not because they believed it was the most important place. Actually it was partly tradition and partly a curious way of emphasising human mediocrity : neither close to heaven or hell, or as Terry Pratchett put it, "the place where the falling angel meets the rising ape."

Yet they did have some truly weird ideas. They didn't believe in parabolic trajectories : they thought that arrows kept going in a straight line until they ran out of energy. They thought the equator was uncrossable because of a ring of fire around the world. And they also believed in evil spirits, witches, and demons.

Science is the means by which we exorcise this demon-haunted world, as Carl Sagan so eloquently put it. We have no fear of crossing the equator any more, or that lonely old women might turn us into frogs or eat our children. We aren't afraid of fairies or goblins or werewolves or dragons. We don't throw human faeces into the street and we don't try and bleed people when they're unwell, or drill holes in their heads, or give them something to bite on while we saw their legs off. And we don't enslave people on the grounds that they are sub-human.

It's a shame about the dragons though.

That is the price of ignorance, and it's a high one : a world of magic and miracles, rituals and superstition. When knowledge is not required, there is no check or limit on stupidity. Even in modern times, there is no limit on how stupid even very intelligent people can become - but the safeguard is that we no longer have a handful of intellectuals trying to solve problems, we've got thousands.

When you insist that something isn't worth knowing, you allow truth to be replaced with doctrine. "No, of course the other side of the world isn't inhabited, how could anyone have crossed the ring of fire at the equator ?" You don't know what the world is really like unless you check. The ancient Greeks didn't believe in atoms because the space between them would be a vacuum, which "obviously" couldn't exist. The truth is that there is absolutely no compunction for the Universe to be what a blood-soaked lump of goop thinks it should be.

You could say to the Universe, this is not fair. And the Universe would say, "Oh, isn't it ? Sorry." - Terry Pratchett, Interesting Times

If you think some piece of research is "obviously" wrong or useless without having studied it, you are a moron. You are here at the behest of the Universe, not the other way around. You do not know the truth because your intuition tells you it. You have the right to your opinion, but fortunately for the rest of us your opinion doesn't count for turnips unless you produce actual evidence.

Even when knowledge doesn't produce direct, tangible results, it can still bring great benefits. Knowing that earthquakes are natural phenomena hasn't helped us predict them yet, but we no longer fear them as the will of the gods. We aren't sacrificing people any more whenever catastrophes threaten. Sometimes, knowing what doesn't work is at least as valuable as knowing what does.

This, then, is the value of knowledge for knowledge's sake : preventing ignorance and all its terrible consequences. Knowledge isn't a an impenetrable shield against fear, because humans are irrational, but it's is a pretty effective one nonetheless - and a damn sight better than the alternative.


Clarity Without Certainty

"Knowledge" is a loaded term. Actually what it's really all about is not knowing facts, but thinking. Simply thinking. "I think this is true" rather than, "I know this is true". Knowing facts is only the beginning - being able to derive Maxwell's equations in your sleep won't help you one bit to be compassionate. But it is a vital first step. Facts lead to models about how the Universe works, allowing you to make predictions and therefore the really interesting part : choices.

I recently attempted to explain to someone that science doesn't claim a monopoly on truth, that it simply tries to get the best information available at the current time. They were quite unable to understand that science does not - almost all of the time - deal in certainties. It only deals with the most likely explanations given the available evidence. Emphasis on "most". Science can't tell you what the correct decision to make is, but it can help you (and only help you) make the best decision possible at the time.

Certainty can even be dangerous. Frank Herbert once wrote that, "Knowledge is the most perfect barrier against learning." Actually, it's certain knowledge. What science provides is not unchangeable knowledge, it's information. Without evidence to disprove it it's as unyielding as stone. With firm evidence against it melts away like... umm... butter ? Yeah. Slowly, and it's quite sticky, and some people would rather it didn't go at all, but ultimately it's gone.

There have been many martyrs who believed in the certainty of their cause. One of my long-standing heroes is Socrates, the philosopher who died not for knowledge but for the right to think, to question, to admit ignorance where it exists and not proclaim false certainty, in short - to doubt. Doubt, in moderation, is not a weakness - it is the most potent weapon available in the arsenal of human intelligence. It is the force that has taken us from an era in which we were certain that we should bleed people who were half-dead into one in which the only certainty is that we can create better and better treatments. It is the means by which we improve ourselves and the world around us.

And so I ask again : what is the value of learning useless information ? What worth is there in learning for the sake of learning ?

Sense and Sensible Statistics

Recently I wrote about the value of the humanities classes in teaching rational thinking. Understanding why something evokes an emotional response is critical to understanding the intent of an article, i.e. how the author is trying to manipulate you. And once you have that, you can begin to step outside the emotions and think logically.

But that's only the beginning. While some of the humanities courses in schools are very good, what we really seem to be lacking almost entirely is a good course on statistics for the under 16's. Because if you take a statistical view of the world, then it's really not as bad as you might think.

Of course I don't mean that we need more emphasis on teaching mathematical techniques - yes, it's important to understand the mean and median and why the median is usually the better, but those are hardly the most important aspects. What I'm on about are the much less mathematical, more philosophical aspects of statistical measurements. You don't need any maths to understand them, but I see a great many people who just don't seem to get them at all. Yet while they're really a lot simpler to explain than the mathematical aspects, they're probably even more important. So, in no particular order :


Anecdotes Are Not Evidence

I just said, "I see a great many people". But what does that mean ? How many people ? Where ? Did I actually talk to them in detail or just form a snap judgement based on one short quote ? You see, by itself, my observation that I personally have witnessed some number of people not understanding statistical methods proves precisely nothing about how many people overall really do not understand statistical methods. Without more details (which I'll get on to in a minute) there could be any number of reasons why my casual observation is meaningless.

But anecdotes aren't unrelated to evidence. For statistics they're a sort of base unit of evidence. As long as the witness isn't lying or delusional, they do prove individual things happen. The problem is that my statement, "I see lots of people who..." strongly implies that I think there's a majority of people who behave in a certain way. And I might think that. I might very well think that. But as to whether it's really true...

Ian Richardson is infinitely more charismatic than Kevin Spacey and anyone who disagrees is objectively wrong.

All my observation can tell us is that some number of people don't understand statistical methods. Sometimes that's good enough : "I saw that man attacking that adorable kitten, officer !". You can use anecdotes to refute sweeping, overstated generalizations ("All men attack kittens.", "Ah, but I knew a man who never attacked a kitten at all !"), but not to support generalizations ("It must be true, I've seen lots of men attacking kittens"). As an estimate as to what fraction of men really attack adorable kittens, it's useless. It's extremely difficult to overcome a personal bias when we witness something happening first-hand for ourselves, especially if we see it repeatedly. Which brings us neatly on to...


Selection Effects

It's almost impossible to get everyone's opinion on any topic or analyse everything in any sample, which is why we need statistics in the first place. But questioning as many people as you can could be utterly useless if you question only specific people. If you're trying to find out how many people enjoy reading, you don't go and only ask people visiting the library, because that's just plain silly.

Of course, you can't ignore the evidence of your own eyes. You see youths being aggressive day in, day out, and it's easy to conclude that young people are aggressive. And it's even true in your experience. But if you're walking the same route each day and see the same youths, you've limited your sample. Or you might be going through a park where the local ruffians choose to congregate, so you're only seeing the dregs and not the far greater numbers of young people who are busy in school.

It's very difficult to eliminate all selection effects when collecting data. But if you don't try to do this at all, you'll end up with a very warped view of the world. Which is probably why people seem to think that most immigrants in Britain are Polish plumbers, even though they're actually highly skilled twenty-somethings.


Think Of The Big Picture

Statistical thinking cautions us to remember that while what we observe is always true, it isn't necessarily representative of what's going on everywhere. Maybe most teenagers in parks are hooligans, but most teenagers overall are just lovely. The point is that you have to be very careful about generalizing from specific observations. The more data you've got, and the wider variety of sources it comes from, the better.

For example, the media often focus on stories about individuals. Being basically empathetic creatures, we react strongly to emotional, personal stories. The trouble is that it doesn't matter how many "violent immigrant" stories you report, all you're doing is picking out anecdotes without reporting the full story. It's a bit like only reporting plane crashes - obviously, there's no story when a plane doesn't crash, because people aren't interested. But we all know planes are safe enough, we accept the small risk that comes with flying because we're at least broadly aware of how many planes don't crash.

It's harder to escape the emotional impact of a violent, personal attack. Our pattern recognition skills tell us, "That dangerous person is in some way different, therefore any people which share that difference might also be dangerous." But unless you also consider the number of such people who aren't dangerous, you are not thinking statistically. You also have to consider how people in other groups behave, otherwise you have nothing to compare with. For instance, 98% of all terrorism in the Western hemisphere is carried out by non-Muslims - which is hardly the view one gets from the media*. People like fear because it stops them from the more difficult task of actually thinking. That sells newspapers but it doesn't tell you what's really going on.

* Strictly speaking if you want to find out if Muslims are more violent, you should look at how many violent acts occur per capita by Muslims compared to all other demographics.

It also puts into perspective how some awful crimes just really aren't that much of a problem. Statistically, terrorism causes about as many annual fatalities as pregnancy, and no-one is talking about a "war on babies".

Underlying Causes

Your local observation of gangs of young ruffians consistently appearing in a local park tells you that you should be wary of those ruffians in that park. It does not, by itself, tell you that you should be wary of all parks or all youngsters. What if that park is in a city with a very high crime rate anyway ? It could be that you're seeing kids in a park because kids hang out in parks; that they're also violent could be related to the fact that the whole area has a high rate of violence in all age groups. There's nothing wrong with your specific knowledge of the area, but you're jumping the gun to assume that all parks (or all youths) are the same everywhere.

It's easy to see why we think this way : we have monkey minds in a modern world. It makes sense to run away from all tigers, because all tigers really are dangerous. The trouble is that we try to apply this thinking to far more complex, modern situations, and it's failing miserably. Instead of making us safer, it's making things more dangerous - our unfounded fears about certain groups cause us to hate them, which causes them to hate us, and the cycle of hate and violence can be difficult to break.

But even when your specific observation is borne out in more general trends, that doesn't necessarily mean anything either. Even if you did see that all kids in parks were violent, it would be silly to conclude that parks make them violent. Similarly it's plainly ridiculous to say that being tall and skinny is a sign of intelligence. Doing intelligent things, like completing a degree in mathematics, is a sign of intelligence - that you're a sexy partygoer is completely and utterly irrelevant, and really quite insulting.

The excellent Spurious Correlations website is full of examples of this, although my favourite has to be this one :

Does eating more chocolate increase your chances of winning a Nobel prize ? Probably not. First, we could turn it around. It doesn't make a lot of sense to say that a few academics winning Nobel prizes causes the whole populace to eat more chocolate, so you can't assume the reverse is true. There could be any number of common reasons why both chocolate consumption and Nobel prize winnings increase simultaneously. In poor countries the population are starved of all foods, so they aren't healthy and have little time to spend on science, while the reverse is true in richer countries. There's a very strong selection effect at work here : why only look at chocolate ? What's the correlation like with other foods ?

Determining what the underlying cause really is is difficult. Ideally you perform an analysis where you see how one variable correlates with lots of other variables, not just one. If you find a correlation and there's a physical mechanism to cause it and no other variable seems to correlate as well, then maybe you've found something interesting.


Ask ALL The Questions !

If a study is focused on a very narrow area, you might think that you can get away with asking very short, simple questions. Not necessarily. You might be introducing a selection effect and miss something very important that's going on. If you're monitoring library usage and find that it's dropping, you don't just ask people whether the chairs are uncomfortable. Ideally you want as much data as possible, so that you can consider both causes you consider likely and unlikely on an equal footing.

Then there's the hugely complicated topic of asking the right questions in the right way. I'm not going to go into this one save to mention this brilliant Yes Minister scene which shows why it's so important :

You've also got to ask the same questions, even if they're not ideal. If you ask people, "Do you like pork pies ?" in Hull and, "Are pork pies your favourite food ?" in Doncaster you will inevitably get different results. This is an even bigger problem when it comes to international studies, since different countries don't always cooperate to get public opinion in the same way. For example, recently there was a claim that America is a less violent place than Britain, which is revealed as pure nonsense when you realise that the two countries have very different definitions of violent crime.



Outliers Are More Noticeable

Selection effects are constantly at work in human memory. We only notice events, we don't notice non-events. A plane that doesn't crash isn't memorable. An immigrant who never breaks the law doesn't stand out. Negative outliers are perhaps even more memorable, because it's safer to remember danger than it is to remember the examples of success. The thing to remember with media stories is that in general, stories only make the news because they're unusual. For that reason, be extremely wary of judging whether anything the media is reporting is typical of what's usually going on. And also be acutely aware that because of this, the media are feeding you a series of unusual events, which will inevitably bias your memory and impressions of what usually happens.

Oddly enough, while it's always possible to point to events in which people were killed, it's not always possible to say when lives are saved - at least not specific, personal examples. It's easy to say when someone dies of heart disease. It's impossible to point to individuals who never get heart disease in the first place because, say, of changes in food regulations or campaigns for healthy eating.

One of my schoolteachers taught me a classic example of what happened when the British government decided to stop moving the clocks back an hour in winter. Campaigners said this would prevent unnecessary deaths in the evening (i.e. schoolchildren walking home in the dark getting hit by cars). And it did. There was also an increase in the number of deaths in the morning, but it was less than the decrease in the evening. So statistically, lives were saved.

But was the media full of stories of children who were, inexplicably, not dead ? No, because you have no idea who was saved by this, but it's very easy to find examples of children killed in the morning, when it was now darker. Of course, you really have no idea who exactly was killed as a result of this either - they might have been run over anyway. Really it makes no more sense to interview the parents of one dead child and say, "this is an example of this law killing children" than it does to to interview the parents of one child who's not dead and say, "this is an example of the law saving lives".

This kind of statistical thinking can seem cold, even cruel and inhuman. In situations like this it's important to remember that we're dealing with probabilities and risk, not individuals. You might think it's a choice of saying, "I want to kill lots more children in the evening and a few less in the morning" or "I want to kill a lot less children in the evening and a few more in the morning", so that basically it boils down to how much killing you want to do - you do not have the luxury of a good choice here.

It's true, but of course altering risk is not the same as either lining up people for a firing squad or rescuing them from a hungry shark. You'll never know who was saved and who was not - you have to go on the numbers, because that's all you've got. You can't avoid taking risks. You can only control which risks you take.


Significance

Unlikely events still happen by chance. If something has a 1 in 10,000 chance of occurring, you can expect that it will occur if the requisite scenario actually does happen 10,000 times. So if you get 10,000 people to flip a coin ten times and one them comes down heads ten times in a row, it's not because one of the people was psychic.

Measuring how statistically significant an event is - the probability that it didn't happen by chance - can be mathematically complicated. It also relies on accounting for those all-important selection effects. Be especially wary of the phenomenon of small number statistics. The smaller your sample, the greater the chance that it can appear to show a trend where actually none exists. For example, if you ask a million people if they like cheese and 600,000 say yes, the result is far more decisive than if you poll ten people and nine of them said yes.

The point is that you should be cautious when an unlikely event happens because it might have happened by chance anyway. Weather prediction is a good example. You can never attribute an individual very hot summer to global warming, because it might have happened anyway. A run of three hot summers is also difficult, because the Earth has been around an awful long time so has probably had umpteen bouts of three hot summers in a row. At what point you can start to say, "this is significant" you'll need to call in a climate expert.

Conclusions

So can these few simple lessons really make you happier ? Possibly. It certainly teaches caution : just because you feel something is true from your own experience doesn't mean that it really is. To be more accurate, personal experience is not a good guide to general trends. Without having experienced other situations, you don't know what selection effects are at work. So anecdotes can tell you something about individual situations, but they're nearly useless when it comes to the big picture. Even your general impressions about what's going on in the world at large are coloured by a media for whom selling emotional but statistically irrelevant stories is de rigueur.

I shall return to the "generation of spoiled idiots" in a moment, but the main point is that we don't often stop to think about how much worse things could be. This is what I call the Grandparent Paradox (nothing to do with the time-travelling Grandfather Paradox). Grandparents are old, experienced people, and experience is valuable. Yet despite having survived World War II, my grandmother was convinced the world was getting worse. Yeah, really, worse than say, being bombed by Nazi Germany, or having to live in a world where women couldn't do equal work for equal pay, homosexuality was illegal, and being an abject racist was just normal.

 "More bad news", she'd say, having just read the latest edition of the Daily "we supported Hitler" Mail. As though the assassination of the Tazenikistani Royal Family (or some such) was somehow impinging on her own need to turn up the heating to a level able to roast a turkey, set the telly on full volume and go off on racist rants for no apparent reason (with hindsight the Daily Mail probably had a lot to do with that).

Statistical thinking means you don't lose your head because something awful happened. Awful things happen all the time, and they probably always will. What they don't indicate is that things are getting worse - you have to look at numbers over time for that. "I see this happening more and more" means precisely diddly-squat without numbers.

Over the last century we've gone from a situation in which racism was normal to, if far from gone, then at least being hated by the majority of people. We're living longer. Our standard of living is immeasurably higher. Women and minorities have equal rights, if not yet equal treatment. And yes, progress hasn't been linear, but inferring that the apocalypse is round the corner because something bad just happened is just dumb.

The other notable quote my gran used to say was that my generation couldn't possibly have fought the Nazis. How she came to this conclusion was anyone's guess, never mind the ridiculous notion about how people "were nicer in those days." It's not my generation that put up signs in pubs saying, "No blacks or Irish". Similarly, the idea that we're a generation of "spoiled idiots" does have a base emotional appeal, but is it really true ?

That's probably the internet meme I hate the most, because every single damn thing about it is wrong. First, I've never seen this in any school playground*. Cardiff, Arecibo, Prague... nope, not happening. Show me the evidence that this does happen and I'll believe you. Second, what's wrong with kids playing with smartphones and not playing sport ? I freakin' HATED sport ! I'd have wounded small puppies for the chance to get out of  P.E. lessons and play with a handheld computer. Why the hell would it have been better for me to suffer ? Kids having access to more advanced technology is a good thing. That's how technological progress is supposed to work.

* Of course, all my anecdote proves is that kids don't avoid sports in every school.

Our continual raising of our own expectations is a mixed blessing. We're continually driven for self-improvement : racism isn't considered normal any more, and homophobia is heading the same way. The downside is that whenever things do get worse we tend to forget how far we've come overall, and because we continually shift the goalposts we're never happy.

Grandparents, in my experience, also seem to have a view that that the modern world is a more dangerous place. "Oh we were allowed to wander off by ourselves when I was six, and we could eat dog poop if we wanted and it never did me any harm", they say. Sure you were. That doesn't mean the world or dog poop was safer (or that medical advice was wrong*) - it could just mean that your parents were stupider, or less well-informed, or didn't love you as much, or just weren't paying attention, or in fact were better at parenting because they let you have more fun. The idea that the world has become more dangerous is only one possible explanation, and just because you see a lot more media reports about murderers doesn't mean there are more murderers around.

* Yeah, you may have gotten away with not washing your hands once in a while. Cleanliness doesn't guarantee survival any more than dirtiness guarantees death - all you're doing is altering the risk. Some changes can't be seen on an individual level - they require a much larger statistical view. There's also a selection effect here : of course it did you no harm, because those people who it did harm are dead and can't complain about it.

The main lesson of statistics isn't that you should jump around in some kind of orgasmic ecstasy about how great the world it, because large aspects of the world are pretty darn crappy. But it reminds us that while we don't live in a Utopia, we do live in something that's infinitely better than many alternatives. We've escaped warlords and petty kings and absolute monarchies and emperors, serfdom and slavery and the workhouse. True, the world isn't what it once was - the only constant is change. But in so many ways, that change has been for the better. I think if we grew up learning about these basic statistical effects, we'd all be a lot more rational. And maybe - just maybe - we'd all be a little bit happier too.

When Worlds Collide : Science In Society

What Is Science, Anyway ?

Science, someone told me recently, is a business. This bold assertion made me too angry to respond because I've seen first-hand what happens if you try and run an observatory as a business, and it just plain doesn't work. Science is not a business. Nor is it art, or politics, or journalism, or a religion, or anything else. It's it's own thing, but like most enterprises it does have some aspects of all of those. Today I'm going to explore a few of those and look at how science and other endeavours relate to each other.


Is science a religion ? Actually screw this one, I can't be bothered.

I've written far too much to bother repeating myself as to why science isn't dogmatic to go through all that again. Simply put, if you think science clings fervently to its beliefs, you are deluded. Nor does religion refuse to admit its mistakes either, though both religious fundamentalists and individual, highly intelligent scientists can and do sometimes behave like this. No amount of devout atheism will save you from being a blithering idiot - or, more to the point, from having any irrational beliefs at all. But I really don't want to dwell on this today, so just go and watch the movie Contact. Don't worry, I'll wait. Just don't bother bringing this up in the comments because I'm not in the mood, OK ?

Fair enough. What about the arts ?

You might not necessarily think that science has much in common with art either, but you'd be wrong. As explained previously, the humanities courses can be an essential tool in developing rational thinking. I'll get to the most obviously relevant form of this - journalism - in a minute, but even the purely creative arts share important aspects with science. And I don't mean the more obvious overlap of scientific illustrations, which have to be able to inform as well as (in a sense) entertain - I mean something much more fundamental.

Problem solving is not so very different to creativity. This is especially important in theoretical physics, where thinking up radically new concepts is the key to making breakthroughs, but it's also essential to be able to come up with new ways of analysing data or doing new tricks with old technology. Finding new interpretations of old data is also tantamount to being creative.

These similarities mean that sometimes the process of doing both science and the arts can be very similar. Both require large amounts of time to do nothing but thinking (and in the case of science at least, an awful lot of background reading). Inspiration can't be forced - you cannot make people have new ideas. You can, however, encourage them. Science and art are both sometimes highly elaborate forms of play, to explore the question, "what if I did it this way...", or better yet, "what does this button do ?" Such thinking intrinsically demands a liberal, reasonably informal atmosphere. Insisting that people are at their desks during some particular set of hours and only talk to each other during scheduled meetings makes absolutely no damn sense whatsoever.

And art and science sometimes both require large amounts of trial and error and sheer patience. Scientific productivity is hard to measure in any case (more on that later), but it simply cannot be evaluated at all on hourly, daily, or even weekly timescales.  Above all, both require the freedom to fail - to spend years on something that may very well be utterly useless without worrying that it might end careers if nothing useful turns up*.

* Recent example - this project took a full seven years to complete, and it could easily have been the case that we would have found nothing interesting whatsoever. So you can take your hourly timesheets and file them where the sun does not shine, thanks.

You might say, "Well, in science a negative result is always useful, and in art, paintings that people hate today might be seen as masterpieces eventually." And that's partially true. Sometimes results in science also aren't recognized as significant until centuries after their inception. And, sometimes, like terrible works of art, they are just god-awful and done by people who are simply very, very stupid.

On the other hand there are plenty of pictures of Jesus, but very few that could be described as "a very hairy monkey in an ill-fitting tunic", so maybe this one wasn't such a waste after all.

The point is that sometimes there's just no way to know in advance if your research will be a load of tripe or lead to the invention of electrical power. To invest only in research you believe is more likely to have practical benefits is the sign of a small mind bereft of vision. You cannot innovate without taking risks. With science, the payoffs - electricity, radio, television, communications satellites, hurricane prediction, the internet, fewer diseases, greater food production, cheap global travel - are so stupendously important that I'm continually amazed by the ongoing need to justify science expenditure at all. It just makes no sense to me whatsoever. Perhaps it's because these things are such an integral part of our daily lives that we don't often think of them as scientific advances.

And no, I don't care about the typo.

Of course science isn't quite the same as the arts. Scientific freedom of thought has to be constrained by observational evidence, whereas the imagination of an artist has no such limits. Scientists can question whether the observations were done correctly, but that's as far as you can go - if your theory predicts that a plane will fly and it doesn't, well that's tough on you. Art on the other hand has no such clear objective measurement.


Journalism then.

Good journalism is the search for truth, no matter how unpleasant it may be or how many people want to stop you from exposing it (excepting perhaps cases where that might endanger innocent people). Similarly, good science is about the search for truth no matter how crazy it might seem or how opposed it may be to any and all ideologies.

Bad journalism, on the other hand, is all about making a sale : telling people what they want to hear or what evokes such a strong emotional reaction that they can't help but feel it's correct. It's much more about who than what or why. Similarly, bad science is about only going for easy, non-controversial results, never considering alternative ideas, interpreting the evidence to mean what you already think it should mean, and deliberately trying to agree or disagree with specific people rather than their ideas.

Sorry Augustine, but that's total rubbish. If it was true we wouldn't have people still claiming the Earth is frickin' flat or refusing to use vaccines. Maybe natural selection will kill off those idiots eventually, but it's a slow and unnecessary process. Bad journalism and bad science (and bad science communication) can be immensely damaging practises.

Good science is a lot like good journalism. The major difference is that science isn't about people, it's about specific things and general trends : the ability to say, "if I do this, given these conditions, then that will always happen." Journalism, on the other hand is usually only about discovering what happened in the past in a very specific circumstance, rather than what might happen next. Journalist's predictions are (often simply because the number of variables is so large) only speculation, even if they are very well-informed speculation, whereas established scientific results are always true. Heating up lead to ~350 C will always melt it - unlike in politics, it doesn't matter what the mood of the experimenter was or whether they'd gained enough support of their peers.

There's also a curious difference between science and journalism when it comes to impartiality. It's usually the job of a journalist to communicate the findings of experts to the public rather than present their own opinions, at least when it comes to making predictions. But it is not always the job of a scientist to be impartial. Scientists are supposed to be objective, which is not the same thing.

Suppose some imbecile of a politician decides that astrology really works. It's the job of a journalist to interview both the politician, scientists, and possibly members of the public to inform people what those groups believe, not to decide who's right and who's wrong. But it is exactly the job of a scientist to decide what's right and what's wrong, to tell you what the evidence says. When the case is a decisive as this, being impartial is not being objective at all, because astrology objectively doesn't work. It doesn't matter if you disagree, you're wrong. Science is not a democratic process.

Not quite true : you do need people's opinions on facts, but we'll get back to that soon.

It's the job of a scientist to form an opinion based on the evidence. As usual it all comes back to this : the beliefs of science are evidence-based and provisional. But for established results which have mountains of evidence (or even irrefutable proof) backing them up, objectivity and impartiality could not be further apart : a scientist who believes the Earth is flat is an oxymoron.

When it comes to cutting-edge research, however, it's another story entirely. Scientists should, of course, try and assess the evidence in an unbiased way. But that does not mean they shouldn't form an opinion about it, only that they should be prepared to change that opinion given different evidence. Of course, when there is very good opposing evidence to an idea, scientists should be honest enough to state, "this is my opinion, but for an alternative you should talk to that guy." But you cannot expect scientists not to express their opinions at all, which brings us neatly on to the next area.

Keenan's Ring

Or, Galaxies Behaving Badly...

I've been itching to write this post for a while. They say the first one's always the hardest, and that's certainly true of astrophysical papers. Much toil has gone into this over the last year or so on the paper alone, but the results are not only very interesting, they're also pretty.

If we could see the hydrogen gas in the galaxy M33, it would appear around five times larger than the full Moon.

The super-short version is that we've found a new hydrogen cloud (not shown above, you'll have to keep reading for that) near to the Triangulum galaxy. It's the largest, most massive cloud that's been found there in over 30 years, and it's ring shaped. We have no idea why it's a ring, let alone what the cloud actually is. A giant starless galaxy ? A huge shell of hydrogen somehow thrown out of the galaxy ? Nothing seems to work. But to understand why this is so difficult to explain, read on.

(Note that I could have given this the clickbaiting title of "Arecibo Scientists Baffled By Giant Hydrogen Cloud" or, "Find Out Why Astronomers Can't Explain This Huge Celestial Ring" or even, "One Young Scientist Went Looking For Galaxies And You Won't Believe What Happened Next". But I didn't, because I am not a moron.)

As usual, the first part is an introduction. If you'd rather skip straight to the new results, scroll down to the "Hydrogen Hunting" section.


Trouble in the Neighbourhood

At first glance, our Local Group of galaxies is a somewhat dull place. Our Milky Way is like a great galactic introvert, hanging out with just a couple of close friends (Andromeda and Triangulum - also known as M31 and M33) and steering clear of heaving nightclubs like the Virgo cluster. Or so you might think.

The Local Group (left) compared to the nearest rich galaxy cluster (Virgo, right). The field of view - about 6 million light years - is the same in each case; the galaxy sizes have been exaggerated by a factor of twenty. Galaxy images come from the Sloan Digital Sky Survey. Small galaxies in the Local Group are shown as faint transparent fuzzy patches. You can just about see Andromeda flying past, but good luck spotting the tiny Triangulum.

But look a little closer and the Local Group turns out to be less of a fancy soiree and more of a rowdy house party. True, there are only three giant galaxies, whereas the Virgo cluster has many hundreds (possibly thousands). If you count smaller (dwarf) galaxies, the Local Group still only has around a hundred at most - it's nowhere near as crowded as a cluster, but we haven't got to the good stuff yet.

I've already described how the numbers and orbits of those dwarfs aren't anything like what we expected (have a look here, points 5 and 6). In short there aren't anywhere near as many dwarfs as models predict, and while they should be buzzing around the giant galaxies like a swarm of bees, they're actually orbiting in loose planes at right-angles to the discs.

So the Local Group is maybe like some sort of non-conformist hipster club where no-one tells anyone what to do. But that's just what we see in the visible light. When we look at the atomic hydrogen gas* using radio telescopes, things get even stranger - and messier.

* See link for details, but remember that atomic hydrogen can only be seen using radio telescopes - so colours are false.

Two major features dominate the group. The first, and most spectacular, is the Magellanic Stream shown above - a huge stream of hydrogen gas that runs halfway around the sky. Possibly more. Whenever observations get more sensitive, it seems we find it's even longer than we realised.

Most - possibly all - of this gas seems to have come from the Magellanic Clouds, two nearby dwarf galaxies. Exactly how the gas got into this peculiar configuration is not well understood, but probably has something to do with how the two galaxies are interacting with each other whilst orbiting the Milky Way. The stream isn't spectacular just because it's close though. At well over half a million light years, it's one of the longest hydrogen streams known.

Fading between an optical image of the Magellanic Clouds and the hydrogen data which shows the gas stream.

The second feature is another, much fainter stream linking Andromeda and Triangulum. This is another dramatic structure at 850,000 light years long. It's also one of the faintest hydrogen streams known, and was only detected by a survey of exceptionally high sensitivity and low resolution. So, unfortunately, there aren't any pretty pictures of this one.

Map of the M31-M33 stream from a paper by Braun & Thilker 2004.

So whereas the Local Group has three large galaxies and two hydrogen streams which are nothing less than awesome in extent, the Virgo cluster may have thousands of galaxies but it's only got four major streams. Looks like it's a win for the hipster house party, then. Of course the Virgo cluster has its own weirdness, but that's another story.

To understand the new results, there's one other important feature of the Local Group you need to know about. Although the main part of the Magellanic Stream doesn't go anywhere near Triangulum, a bunch of smaller clouds do. Could there be some connection between the Magellanic and the Andromeda streams ? I dunno. M33 and M31 are a lot further away than the Magellanic Clouds and the stream would have a really weird, sharp kink in it, so it could just be a complete coincidence.

Probably the best map of the stream produced to date, from a paper by Nidever et al. 2010. M33 is just above Wright's Cloud (which we'll talk about soon).

Hydrogen Hunting

Now that we've got the proper context, the new results. Most of the smaller galaxies in the Local Group are orbiting the Milky Way and Andromeda. Only one is believed to be associated with Triangulum, and that's not certain. Together with all these crazy streams (especially the M33-M31 stream), back in 2006 it seemed like a good idea to look at the hydrogen in and around M33 as part of the Arecibo Galaxy Environment Survey, AGES. This is one of the most sensitive atomic hydrogen surveys ever performed, so if there's anything interesting, AGES is what will find it.

"If all else fails, point a 300m telescope at it."

It took us five years to complete the observations - without any guarantee that we'd detect anything interesting in the end. That's the problem with a telescope that can't be steered much, but it isn't normally this bad. The difficulty was that the survey area is at the absolute northern limit of where Arecibo can see, so often we could only observe it for an hour a night. Factor in that other projects sometimes get priority, the fact that we can't see the region at all for more than half the year, and various equipment failures, and five years doesn't seem quite so bad.

Five years of stating at computer screens...

More staring at screens...

Everybody look serious now. Come on people, this is science.

Occasional bouts of madness began to set in...

... which were usually solved with late-night Dominion sessions.

Sometimes even the ALFA receiver went a little nuts and had to be taken down for repairs.

Most of the people above didn't do observations of M33. In fact most of the time we just did it by setting up a script and going home.


After all this, what did we find ? Well, firstly that the hydrogen of Triangulum is considerably more extended than what the last major survey of the area had revealed. Some of the "clouds" that ALFALFA had found turned out to just be slightly denser parts of this larger disc. Also, the disc is really quite faint - so faint, according to earlier work, it should be ionized. But it isn't. You could call it, "The Hydrogen Cloud So Large It Shouldn't Exist", if you want to clickbait it*.

* But don't. It's not the only hydrogen detection known below the ionization threshold nor the first such discovery.

Animation showing the hydrogen in M33 at different sensitivity levels. If your sensitivity isn't very good, all you'll see is the densest gas in the stellar disc. Go a bit deeper and you'll see things are a lot more extended, and part of the disc (upper right) is warped. Go as deep as AGES and you'll see that even this extended component is actually just part of something even larger.

There's one more brief digression I must make for the sake of non-regular readers. Hydrogen observations depend on frequency - that is, how fast something is moving towards or away from us. Which means our radio maps are actually 3D data cubes. They're very cool to look at (and extremely useful, as we shall see), but velocity is not the same as distance. For example some of the hydrogen in M33 has the same velocity away from us as does the hydrogen in parts of the Milky Way. So in these data cubes it can look as though the hydrogen in M33 and the Milky Way overlap each other - but in reality they're actually separated by over a million light years.

M33 is the big bright thing in the middle. Some hydrogen from the Milky Way can be seen as the big flat thing at one edge of the cube.

Anyway, although the new observations showed that some clouds weren't clouds at all, they also found new ones that pretty much exactly balanced out the numbers. Just enough to agree with how many satellite galaxies models of galaxy evolution predict there should be... but none of them have any stars !

Could these be the "missing satellites" that people have been searching for for so many years ? It's not impossible, but I doubt it. Firstly, it doesn't seem very likely that only this one galaxy happens to have the right number of companions and they just happen to be made of gas rather than stars - the model's so badly broken for other galaxies, fixing it with this one wouldn't really help. Secondly, the motions of gas in the clouds don't match the model predictions*. Thirdly, they're not distributed like the predicted "swarm of bees" at all - they're found in a distinct band. There's a hint that some of these clouds are actually linked together and part of a much larger structure.

* This is a debatable point though. More precisely, models predict how fast the gas should be moving within galaxies, and also how it's moving - that is, they predict it should be rotating. The observations say the motions are random. But a new paper has just been submitted saying that maybe the motions should be random after all.

Some of the clouds we detected with AGES (M33 is right in the middle, but at a different velocity so it's not visible here). Notice that there aren't any clouds in the top half of the survey area. Though it is awfully strange that the number of clouds is almost exactly enough to account for the missing satellites.

And two of the clouds are much, much bigger than the others. One of them - Wright's Cloud, which we mentioned earlier - has been known about since 1979. The most popular explanation is that it's part of the Magellanic Stream, or at least related to it in some way. Although it's not that close to the main stream, it does seem to follow those smaller clouds heading from the stream towards M33. It's a bit strange though, since Wright's Cloud is far larger than any other cloud at that distance from the Magellanic Clouds.

Now, 36 years later, we can reveal another giant cloud right next to Wright's Cloud that no-one had ever detected before. Well, that's not quite true. Parts of this new cloud had been found by ALFALFA. But our new observations were more sensitive over a larger area, and they've revealed that these clouds are actually just a small part of a much larger ring :

The centre of M33 (not shown) is just above and to the right of the ring.

We're calling this Keenan's Ring after "Princess" Olivia Keenan who discovered it and did most of the hard work of writing the paper. There isn't really any convention for naming hydrogen clouds, but if we get this in common usage it will probably stick. After all, it sounds a lot better than its catalogue designation AGESM33-31.

Map of all the clouds in the area. Keenan's Ring is at about as large as M33 and yet no-one knew about it until now.

So what the heck is it ? It all depends on how far away the cloud is, which we can't measure directly. If it's as far away as M33, then the thing is about 60,000 light years across - as big as the hydrogen of M33 itself ! And that would be seriously strange, because there's no obvious way to form a stonkin' great ring of hydrogen gas like this. Models show that you can do it through colliding galaxies, but M33 doesn't look as though it's had any kind of major collision recently. The gas in the Ring appears to be completely separate to the gas in M33.

The full data cube. Colours are arbitrary - blue was chosen to highlight the gas in the Milky Way, which fills the entire field of view. Orange shows everything else. M33 is the big orange blob in the middle. Wright's Cloud is the brightest-looking orange cloud which is cut off by the edge of the survey area. Keenan's Ring is visible, as are a bunch of other smaller clouds. The data looks noisier around the Ring and the Cloud, but this is just an artifact due to how the data was visualised.

Because this gas cloud is so dang large, could it be a giant, failed galaxy ? A tempting idea. Although M33 doesn't seem to have collided with anything recently, there is a strange "warp" in its hydrogen disc, and Keenan's Ring is on exactly the opposite side of the galaxy. Which is what you might expect if a giant object had come sailing past. Also, the densest gas in the Ring is closest to M33, suggesting that M33 is exerting a gravitational influence on it (which isn't the case for Wright's Cloud, which has a higher density in its center).

But this idea doesn't really work. Although there is a small velocity gradient (the gas on one side is moving at a different speed to that on the other) across the Ring, which is often a sign of rotation, it's far smaller than for M33 (~30 km/s compared to ~180 km/s). A galaxy as large as M33 really ought to be rotating as fast as M33 - if it isn't, then it isn't as massive, so it's far less likely to explain the warp. And if it was massive, that would make the predictions of the models of how many galaxies there are even worse, not better.

It also doesn't seem a likely coincidence that Keenan's Ring and Wright's Cloud are at almost exactly the same velocity. If Wright's Cloud is really just another part of the Magellanic Stream, then it seems probable that Keenan's Ring is as well. This, in my opinion, is the most likely explanation. But it's not without major difficulties either. Why in the world should there be two massive clouds near the end but significantly offset from the main stream ? And if it's not a coincidence that Wight's Cloud is part of the Magellanic Stream because it's so close, then surely by the same token it's at least equally likely to be associated with M33 in some way as well ?

No-one seems to have any answer to this. And it's also worth remembering that Wright's Cloud and Keenan's Ring are at different velocities to the other clouds detected in this area, suggesting that they might have different origins. Yet they're also very different to each other. Wright's Cloud is larger, much more massive, doesn't show any signs of a velocity gradient, has an irregular structure, and is denser in the centre. Keenan's Ring does have a velocity gradient (albeit a small one), is much denser on one side than the other, and is of course ring shaped.

And perhaps the most difficult question to answer applies to any scenario : why is it a ring ? A disc, well, that's fine - could be a giant galaxy, or just a cloud, whatever. An amorphous blob like Wright's Cloud - yeah, also possible, could be interacting with all the other clouds and M33, no problem. But there's no obvious reason why the gas in the middle should be missing. Where's it gone ?

Could it just be that the "ring" is a bunch of clouds which happen to line up and look like a ring ? Not likely. That would require an extremely unlikely chance alignment of clouds - there's no reason you'd expect the central region to be underpopulated just by placing clouds at random. And although the velocity gradient is small, it does have one. If it was a bunch of random clouds, you'd expect the velocity of each cloud to be different. But the velocity of the gas across the Ring varies quite smoothly from one side to the other, which you wouldn't expect if it was made of separate clouds. It's possible, but not credible.

Map showing the velocity of the gas at each point in the Ring

Honestly we really just don't know what this is. Supernovae explosion are known to blast holes in the hydrogen, but if it's close to M33 then the hole is about ten times larger than any other known holes. Nor are there any obvious star clusters (massive stars aren't thought to be able to form in isolation), so if it's inside our own galaxy the Ring would be much smaller but probably just as strange. And it would be one heck of a coincidence if this structure just happened by chance to not only be so close on the sky to Wright's Cloud but at such a similar velocity.

Like all the most interesting discoveries, this one poses a lot more questions than it answers. Which, if you ask me, makes it five years well spent.