Philip Tetlock: scourge of the experts

Thirty years ago at the University of Yale, psychologist Philip Tetlock witnessed an experiment which was to inspire his life’s work: an epic 20-year investigation, the results of which are highly surprising and hold important lessons for us all.

The set up of the experiment was not particularly novel: it involved a rat, a maze, some food and a group of students to observe the results. The rat was released into the maze and given a choice to go left or right at a T-junction. Food was placed to the left on 60% of occasions, and to the right the rest of the time. It didn’t take the rat long to work out that it was more likely to be found to the left than to the right, and it soon chose to go left each time, guaranteeing itself a snack on 60% of occasions.

But this wasn’t the real point of the experiment. At the same time the rat was working for its dinner, the students were also asked to guess on which side the food would be found (they weren’t aware of the set 60% ratio of lefts to rights). The students, in their search for a pattern, switched their predictions regularly between right and left, resulting in a success rate of 52%. The rat was thus declared to have greater predictive powers than a roomful of Yale students.

Tetlock wondered if these observations pointed to a wider problem in the human brain’s ability to predict the future, and whether this problem extended to predictions made by professional forecasters. He embarked on a 20 year odyssey to investigate the ability of these experts to anticipate future trends.

Tetlock studied the forecasts of 284 experts who made their living identifying political and economic trends, periodically asking them to make predictions relating to their fields of expertise. Rather than giving them yes/no questions, he asked them to assess probabilities, for example: ‘What are the chances of inflation in the UK rising, staying the same or falling in the next 12 months?’, or ‘Will casualties in the Israel-Palestine conflict fall, rise, or stay the same in the next year?’ He didn’t just ask for their opinions, but studied the basis of their decision and how they reacted when they were proven to be wrong. After 20 years on the project, Tetlock ended up with an enormous database of 82.361 expert predictions.

The project’s findings were astounding. Just like in the rat test, arbitrarily assigning equal probabilities to the ‘worse’, ’same’ and ‘better’ outcomes would have yielded more accurate predictions than those given by the ‘experts’. In Tetlock’s words, a dart-throwing chimp would have been as much help as the paid experts in predicting the future (although please don’t try this at home, kids).

Other studies into the reliability of economic forecasts have yielded similar results, indicating that basing your forecast on a repeat of the result in the previous period would be more accurate than professional forecasts. This is more than a little disconcerting. Even the most independent-minded of us relies on professional expertise to tell us what the future may bring.

What is it that prevents people who know a subject inside out from outperforming rats and chimps in predicting the future? Some clues can be found in Tetlock’s analysis of his results. He looked for what personal characteristics made his forecasters more or less likely to make good predictions. One trend he noticed was that there was a significant correlation with the fame of the person. The more famous they were, the worse their forecasts tended to be. He also found that, beyond a basic working knowledge, the more an expert knew on the subject they were making predictions about, the more likely their prediction was to be wrong.

What lies behind this highly counterintuitive finding? Tetlock suggests that the experts made similar mistakes to the Yale students. The more you think you know, the more you are likely to overanalyse a situation which is inherently unpredictable or subject to far simpler rules than those applied by the sophisticated forecaster. The more an expert has invested in their complicated theories and arcane knowledge, the more likely they are to unnecessarily apply it to their prediction.

Another contributing factor could be Black Swans. The concept comes originally from the work of philosopher John Stuart Mill, but was popularised by Karl Popper in his influential treatise “The Logic of Scientific Discovery”, in which he dealt with the problem of induction. Induction is a process of thinking whereby a universal conclusion in reached on the basis of a limited number of observations. For anyone living in Europe prior to the discovery of Australia and New Zealand, the statement “all swans are white” would have seemed reasonable. If all the swans you and everyone you know and have ever encountered are white, it seems natural to assume it to be a universal fact that all swans are white – until, that is, someone ruins everything by finding black swans in the Antipodes.

In his excellent book, The Black Swan, Nassim Nicholas Taleb expands the idea of the Black Swan to any phenomenon which comes out of the blue, has a major impact, and which, in hindsight, is analysed to the point where its occurrence seems to have been almost inevitable. Example cited by Taleb include 9/11, World War I, and the advent of the PC and internet. To that list could be added the current financial crisis. Taleb argues that such events have a much larger effect on the world than is anticipated by traditional forecasting methods and by our hard-wired perception of the future.

Take as an example economic forecasts for Canada’s growth in GDP in 2010. According to the Bank of Canada, a growth rate of 3.8% is expected next year, a figure which is so incongruous that it looks suspiciously politically motivated. In comparison, the IMF expects growth of 1.6%, which seems more plausible at first sight. But what do these figures actually mean? What assumptions are they based upon? No more bank failures, one more failure, or how many exactly? Is the collapse of GM factored into the figures or not? How would they be affected by another large terrorist attack on North American soil, or an earthquake in San Francisco? How about a tsunami hitting Tokyo, or a bird flu pandemic? They may not be are not very cheery thoughts, but these and a myriad of other inherently unpredictable events and knife-edge outcomes in aggregate make these predictions worth less than the paper they’re written on. Economists may argue that the law of averages cancels out ‘external factors’, but this is patently untrue in the case of events of this magnitude. The economists are either fooling themselves, or just trying to fool us to keep their jobs intact.

Tetlock also noted in his analysis that the approach of the more successful forecasters differed significantly from the least successful. To explain the difference, he used the analogy of foxes and hedgehogs. According to the ancient Greek poet Archilochus, “The fox knows many things, but the hedgehog knows one big thing”. In other words, for all the cunning a fox has, a hedgehog only needs one tactic (rolling into a ball) to frustrate it. The philosopher Isaiah Berlin used the distinction in his celebrated essay “The Hedgehog and the Fox” to categorise thinkers and writers according to whether their work was based on a unified, consistent view of the world, or whether they brought multiple perspectives into their work.

Tetlock, similarly, labeled as hedgehogs those experts whose predictions were based on the application of one big idea, and who did not waver in their beliefs when they made mistakes, but rather made excuses for their failures. Their in-depth specialization in their subject led them to try to force the square peg of their pet theories into the round holes of subjects where they didn’t fully apply, and thus negatively affected their forecasting capabilities.

Foxes, on the other hand, had no grand theory, but instead drew on a broad range of ideas to determine their prognosis, and were always eager to learn from their mistakes. Tetlock found that foxes were far better forecasters than hedgehogs on average. They may have still only had around a 60% success rate (being a fox does not help one predict black swans), but this was significantly better on average than the hedgehogs.

It should be noted, however, that hedgehogs tend to be far more controversial than foxes, and thus get more column inches and air-time. Television, in particular, loves to pit two hedgehogs with alternate world views against each other. It makes a far more dramatic spectacle than two foxes trying to understand each others’ opinions. This helps to explain the inverse relationship between fame and forecasting prowess.

So where does that leave the layperson in need of expert advice? Firstly, it is clear that one should be skeptical about the worth of any opinions coming from so-called experts. If you have a basic understanding of the subject, you can probably do just as good a job yourself. Tetlock recommends following the predictions of those with a proven track record of correct forecasting. As a caveat to this, however, I would say that this only counts if these predictions demonstrate the expert’s ability to be flexible in his or her outlook. For example, the media has lauded those few brave souls who predicted the current financial crisis. At least some of these, however will probably have been right just because they have an extreme pessimistic bias. Relying on such individuals to accurately predict the bottom of the slump almost certainly be very unwise.

The news that experts are little or no better at predicting the future than ourselves suggests that we don’t need them. Do not expect, however, for them to be thrown out on the street any time soon. Even if I had never heard of chaos theory, my observation of the unreliability of long-term weather forecasting would tell me that they are as good as useless beyond a window of about three days. Nevertheless, every day I check the seven day forecast.

It seems that our brains are so uncomfortable dealing with an undefined future that we will will gladly listen to anyone willing to predict it. For some it may be fortune tellers and tarot cards, for others it may be economists or political analysts. Either way, the reassurance of having some expected future to plan for seems far more important to us than the reliability of the forecast itself. Rather than mocking the experts, perhaps we should first look a little closer to home.