Sense and Sensible Statistics

Recently I wrote about the value of the humanities classes in teaching rational thinking. Understanding why something evokes an emotional response is critical to understanding the intent of an article, i.e. how the author is trying to manipulate you. And once you have that, you can begin to step outside the emotions and think logically.

But that's only the beginning. While some of the humanities courses in schools are very good, what we really seem to be lacking almost entirely is a good course on statistics for the under 16's. Because if you take a statistical view of the world, then it's really not as bad as you might think.

Of course I don't mean that we need more emphasis on teaching mathematical techniques - yes, it's important to understand the mean and median and why the median is usually the better, but those are hardly the most important aspects. What I'm on about are the much less mathematical, more philosophical aspects of statistical measurements. You don't need any maths to understand them, but I see a great many people who just don't seem to get them at all. Yet while they're really a lot simpler to explain than the mathematical aspects, they're probably even more important. So, in no particular order :


Anecdotes Are Not Evidence

I just said, "I see a great many people". But what does that mean ? How many people ? Where ? Did I actually talk to them in detail or just form a snap judgement based on one short quote ? You see, by itself, my observation that I personally have witnessed some number of people not understanding statistical methods proves precisely nothing about how many people overall really do not understand statistical methods. Without more details (which I'll get on to in a minute) there could be any number of reasons why my casual observation is meaningless.

But anecdotes aren't unrelated to evidence. For statistics they're a sort of base unit of evidence. As long as the witness isn't lying or delusional, they do prove individual things happen. The problem is that my statement, "I see lots of people who..." strongly implies that I think there's a majority of people who behave in a certain way. And I might think that. I might very well think that. But as to whether it's really true...

Ian Richardson is infinitely more charismatic than Kevin Spacey and anyone who disagrees is objectively wrong.

All my observation can tell us is that some number of people don't understand statistical methods. Sometimes that's good enough : "I saw that man attacking that adorable kitten, officer !". You can use anecdotes to refute sweeping, overstated generalizations ("All men attack kittens.", "Ah, but I knew a man who never attacked a kitten at all !"), but not to support generalizations ("It must be true, I've seen lots of men attacking kittens"). As an estimate as to what fraction of men really attack adorable kittens, it's useless. It's extremely difficult to overcome a personal bias when we witness something happening first-hand for ourselves, especially if we see it repeatedly. Which brings us neatly on to...


Selection Effects

It's almost impossible to get everyone's opinion on any topic or analyse everything in any sample, which is why we need statistics in the first place. But questioning as many people as you can could be utterly useless if you question only specific people. If you're trying to find out how many people enjoy reading, you don't go and only ask people visiting the library, because that's just plain silly.

Of course, you can't ignore the evidence of your own eyes. You see youths being aggressive day in, day out, and it's easy to conclude that young people are aggressive. And it's even true in your experience. But if you're walking the same route each day and see the same youths, you've limited your sample. Or you might be going through a park where the local ruffians choose to congregate, so you're only seeing the dregs and not the far greater numbers of young people who are busy in school.

It's very difficult to eliminate all selection effects when collecting data. But if you don't try to do this at all, you'll end up with a very warped view of the world. Which is probably why people seem to think that most immigrants in Britain are Polish plumbers, even though they're actually highly skilled twenty-somethings.


Think Of The Big Picture

Statistical thinking cautions us to remember that while what we observe is always true, it isn't necessarily representative of what's going on everywhere. Maybe most teenagers in parks are hooligans, but most teenagers overall are just lovely. The point is that you have to be very careful about generalizing from specific observations. The more data you've got, and the wider variety of sources it comes from, the better.

For example, the media often focus on stories about individuals. Being basically empathetic creatures, we react strongly to emotional, personal stories. The trouble is that it doesn't matter how many "violent immigrant" stories you report, all you're doing is picking out anecdotes without reporting the full story. It's a bit like only reporting plane crashes - obviously, there's no story when a plane doesn't crash, because people aren't interested. But we all know planes are safe enough, we accept the small risk that comes with flying because we're at least broadly aware of how many planes don't crash.

It's harder to escape the emotional impact of a violent, personal attack. Our pattern recognition skills tell us, "That dangerous person is in some way different, therefore any people which share that difference might also be dangerous." But unless you also consider the number of such people who aren't dangerous, you are not thinking statistically. You also have to consider how people in other groups behave, otherwise you have nothing to compare with. For instance, 98% of all terrorism in the Western hemisphere is carried out by non-Muslims - which is hardly the view one gets from the media*. People like fear because it stops them from the more difficult task of actually thinking. That sells newspapers but it doesn't tell you what's really going on.

* Strictly speaking if you want to find out if Muslims are more violent, you should look at how many violent acts occur per capita by Muslims compared to all other demographics.

It also puts into perspective how some awful crimes just really aren't that much of a problem. Statistically, terrorism causes about as many annual fatalities as pregnancy, and no-one is talking about a "war on babies".

Underlying Causes

Your local observation of gangs of young ruffians consistently appearing in a local park tells you that you should be wary of those ruffians in that park. It does not, by itself, tell you that you should be wary of all parks or all youngsters. What if that park is in a city with a very high crime rate anyway ? It could be that you're seeing kids in a park because kids hang out in parks; that they're also violent could be related to the fact that the whole area has a high rate of violence in all age groups. There's nothing wrong with your specific knowledge of the area, but you're jumping the gun to assume that all parks (or all youths) are the same everywhere.

It's easy to see why we think this way : we have monkey minds in a modern world. It makes sense to run away from all tigers, because all tigers really are dangerous. The trouble is that we try to apply this thinking to far more complex, modern situations, and it's failing miserably. Instead of making us safer, it's making things more dangerous - our unfounded fears about certain groups cause us to hate them, which causes them to hate us, and the cycle of hate and violence can be difficult to break.

But even when your specific observation is borne out in more general trends, that doesn't necessarily mean anything either. Even if you did see that all kids in parks were violent, it would be silly to conclude that parks make them violent. Similarly it's plainly ridiculous to say that being tall and skinny is a sign of intelligence. Doing intelligent things, like completing a degree in mathematics, is a sign of intelligence - that you're a sexy partygoer is completely and utterly irrelevant, and really quite insulting.

The excellent Spurious Correlations website is full of examples of this, although my favourite has to be this one :

Does eating more chocolate increase your chances of winning a Nobel prize ? Probably not. First, we could turn it around. It doesn't make a lot of sense to say that a few academics winning Nobel prizes causes the whole populace to eat more chocolate, so you can't assume the reverse is true. There could be any number of common reasons why both chocolate consumption and Nobel prize winnings increase simultaneously. In poor countries the population are starved of all foods, so they aren't healthy and have little time to spend on science, while the reverse is true in richer countries. There's a very strong selection effect at work here : why only look at chocolate ? What's the correlation like with other foods ?

Determining what the underlying cause really is is difficult. Ideally you perform an analysis where you see how one variable correlates with lots of other variables, not just one. If you find a correlation and there's a physical mechanism to cause it and no other variable seems to correlate as well, then maybe you've found something interesting.


Ask ALL The Questions !

If a study is focused on a very narrow area, you might think that you can get away with asking very short, simple questions. Not necessarily. You might be introducing a selection effect and miss something very important that's going on. If you're monitoring library usage and find that it's dropping, you don't just ask people whether the chairs are uncomfortable. Ideally you want as much data as possible, so that you can consider both causes you consider likely and unlikely on an equal footing.

Then there's the hugely complicated topic of asking the right questions in the right way. I'm not going to go into this one save to mention this brilliant Yes Minister scene which shows why it's so important :

You've also got to ask the same questions, even if they're not ideal. If you ask people, "Do you like pork pies ?" in Hull and, "Are pork pies your favourite food ?" in Doncaster you will inevitably get different results. This is an even bigger problem when it comes to international studies, since different countries don't always cooperate to get public opinion in the same way. For example, recently there was a claim that America is a less violent place than Britain, which is revealed as pure nonsense when you realise that the two countries have very different definitions of violent crime.



Outliers Are More Noticeable

Selection effects are constantly at work in human memory. We only notice events, we don't notice non-events. A plane that doesn't crash isn't memorable. An immigrant who never breaks the law doesn't stand out. Negative outliers are perhaps even more memorable, because it's safer to remember danger than it is to remember the examples of success. The thing to remember with media stories is that in general, stories only make the news because they're unusual. For that reason, be extremely wary of judging whether anything the media is reporting is typical of what's usually going on. And also be acutely aware that because of this, the media are feeding you a series of unusual events, which will inevitably bias your memory and impressions of what usually happens.

Oddly enough, while it's always possible to point to events in which people were killed, it's not always possible to say when lives are saved - at least not specific, personal examples. It's easy to say when someone dies of heart disease. It's impossible to point to individuals who never get heart disease in the first place because, say, of changes in food regulations or campaigns for healthy eating.

One of my schoolteachers taught me a classic example of what happened when the British government decided to stop moving the clocks back an hour in winter. Campaigners said this would prevent unnecessary deaths in the evening (i.e. schoolchildren walking home in the dark getting hit by cars). And it did. There was also an increase in the number of deaths in the morning, but it was less than the decrease in the evening. So statistically, lives were saved.

But was the media full of stories of children who were, inexplicably, not dead ? No, because you have no idea who was saved by this, but it's very easy to find examples of children killed in the morning, when it was now darker. Of course, you really have no idea who exactly was killed as a result of this either - they might have been run over anyway. Really it makes no more sense to interview the parents of one dead child and say, "this is an example of this law killing children" than it does to to interview the parents of one child who's not dead and say, "this is an example of the law saving lives".

This kind of statistical thinking can seem cold, even cruel and inhuman. In situations like this it's important to remember that we're dealing with probabilities and risk, not individuals. You might think it's a choice of saying, "I want to kill lots more children in the evening and a few less in the morning" or "I want to kill a lot less children in the evening and a few more in the morning", so that basically it boils down to how much killing you want to do - you do not have the luxury of a good choice here.

It's true, but of course altering risk is not the same as either lining up people for a firing squad or rescuing them from a hungry shark. You'll never know who was saved and who was not - you have to go on the numbers, because that's all you've got. You can't avoid taking risks. You can only control which risks you take.


Significance

Unlikely events still happen by chance. If something has a 1 in 10,000 chance of occurring, you can expect that it will occur if the requisite scenario actually does happen 10,000 times. So if you get 10,000 people to flip a coin ten times and one them comes down heads ten times in a row, it's not because one of the people was psychic.

Measuring how statistically significant an event is - the probability that it didn't happen by chance - can be mathematically complicated. It also relies on accounting for those all-important selection effects. Be especially wary of the phenomenon of small number statistics. The smaller your sample, the greater the chance that it can appear to show a trend where actually none exists. For example, if you ask a million people if they like cheese and 600,000 say yes, the result is far more decisive than if you poll ten people and nine of them said yes.

The point is that you should be cautious when an unlikely event happens because it might have happened by chance anyway. Weather prediction is a good example. You can never attribute an individual very hot summer to global warming, because it might have happened anyway. A run of three hot summers is also difficult, because the Earth has been around an awful long time so has probably had umpteen bouts of three hot summers in a row. At what point you can start to say, "this is significant" you'll need to call in a climate expert.

Conclusions

So can these few simple lessons really make you happier ? Possibly. It certainly teaches caution : just because you feel something is true from your own experience doesn't mean that it really is. To be more accurate, personal experience is not a good guide to general trends. Without having experienced other situations, you don't know what selection effects are at work. So anecdotes can tell you something about individual situations, but they're nearly useless when it comes to the big picture. Even your general impressions about what's going on in the world at large are coloured by a media for whom selling emotional but statistically irrelevant stories is de rigueur.

I shall return to the "generation of spoiled idiots" in a moment, but the main point is that we don't often stop to think about how much worse things could be. This is what I call the Grandparent Paradox (nothing to do with the time-travelling Grandfather Paradox). Grandparents are old, experienced people, and experience is valuable. Yet despite having survived World War II, my grandmother was convinced the world was getting worse. Yeah, really, worse than say, being bombed by Nazi Germany, or having to live in a world where women couldn't do equal work for equal pay, homosexuality was illegal, and being an abject racist was just normal.

 "More bad news", she'd say, having just read the latest edition of the Daily "we supported Hitler" Mail. As though the assassination of the Tazenikistani Royal Family (or some such) was somehow impinging on her own need to turn up the heating to a level able to roast a turkey, set the telly on full volume and go off on racist rants for no apparent reason (with hindsight the Daily Mail probably had a lot to do with that).

Statistical thinking means you don't lose your head because something awful happened. Awful things happen all the time, and they probably always will. What they don't indicate is that things are getting worse - you have to look at numbers over time for that. "I see this happening more and more" means precisely diddly-squat without numbers.

Over the last century we've gone from a situation in which racism was normal to, if far from gone, then at least being hated by the majority of people. We're living longer. Our standard of living is immeasurably higher. Women and minorities have equal rights, if not yet equal treatment. And yes, progress hasn't been linear, but inferring that the apocalypse is round the corner because something bad just happened is just dumb.

The other notable quote my gran used to say was that my generation couldn't possibly have fought the Nazis. How she came to this conclusion was anyone's guess, never mind the ridiculous notion about how people "were nicer in those days." It's not my generation that put up signs in pubs saying, "No blacks or Irish". Similarly, the idea that we're a generation of "spoiled idiots" does have a base emotional appeal, but is it really true ?

That's probably the internet meme I hate the most, because every single damn thing about it is wrong. First, I've never seen this in any school playground*. Cardiff, Arecibo, Prague... nope, not happening. Show me the evidence that this does happen and I'll believe you. Second, what's wrong with kids playing with smartphones and not playing sport ? I freakin' HATED sport ! I'd have wounded small puppies for the chance to get out of  P.E. lessons and play with a handheld computer. Why the hell would it have been better for me to suffer ? Kids having access to more advanced technology is a good thing. That's how technological progress is supposed to work.

* Of course, all my anecdote proves is that kids don't avoid sports in every school.

Our continual raising of our own expectations is a mixed blessing. We're continually driven for self-improvement : racism isn't considered normal any more, and homophobia is heading the same way. The downside is that whenever things do get worse we tend to forget how far we've come overall, and because we continually shift the goalposts we're never happy.

Grandparents, in my experience, also seem to have a view that that the modern world is a more dangerous place. "Oh we were allowed to wander off by ourselves when I was six, and we could eat dog poop if we wanted and it never did me any harm", they say. Sure you were. That doesn't mean the world or dog poop was safer (or that medical advice was wrong*) - it could just mean that your parents were stupider, or less well-informed, or didn't love you as much, or just weren't paying attention, or in fact were better at parenting because they let you have more fun. The idea that the world has become more dangerous is only one possible explanation, and just because you see a lot more media reports about murderers doesn't mean there are more murderers around.

* Yeah, you may have gotten away with not washing your hands once in a while. Cleanliness doesn't guarantee survival any more than dirtiness guarantees death - all you're doing is altering the risk. Some changes can't be seen on an individual level - they require a much larger statistical view. There's also a selection effect here : of course it did you no harm, because those people who it did harm are dead and can't complain about it.

The main lesson of statistics isn't that you should jump around in some kind of orgasmic ecstasy about how great the world it, because large aspects of the world are pretty darn crappy. But it reminds us that while we don't live in a Utopia, we do live in something that's infinitely better than many alternatives. We've escaped warlords and petty kings and absolute monarchies and emperors, serfdom and slavery and the workhouse. True, the world isn't what it once was - the only constant is change. But in so many ways, that change has been for the better. I think if we grew up learning about these basic statistical effects, we'd all be a lot more rational. And maybe - just maybe - we'd all be a little bit happier too.