Overconfidence Bias and Organizational Decision-Making

"Often wrong, never in doubt."

The public fascination with pseudo-science is hard to fathom when actual science is so fascinating. (The observer affects the experiment, people!) Nonetheless, horoscopes, psychic readings and personality tests persist, likely because they provide people with a (false) sense of control.

One particular practice became all the rage in the early 2000's: visualization. The idea was that if we simply visualized what we wanted to happen to us, the universe would manifest it. (So, if you visualized hard enough, you'd never have to look for parking again.)

Of course, subscribing to this notion demonstrates a complete lack of empathy, unless you want to argue that kidnapped children forced into military service are terrible at visualization. Regardless, visualization hit fever pitch, and then eventually died down, presumably when people realized that visualizing did not work as advertised.

The problem, of course, is that planning takes a lot more work. And even putting in the work offers no guarantees, merely improved chances of success. There's a reason it's a classic: "The best way to make God laugh is to make a plan." And if there's anything that people hate doing, it's putting in the work with no guarantees of success.

So rather than living with the discomfort of that uncertainty, we adjust our perspective to make things feel more comfortable. That perspective change is called Overconfidence Bias, and it extends equally into what we can and cannot control.

We attribute our success to talent and our failure to luck. We shield ourselves from calamity when we look into our futures. And we either downplay the significance of factors outside our control, or overestimate our ability to handle them.

This is all in service of improving our lived experiences, and it's hard to argue the practice. Life is hard, and it makes sense that we adapt to maximize our experiences with it. No one is advocating that we should, or even could, change in this regard.

However, we also need to recognize that this is happening, otherwise it will have a terrible impact on our decision-making. The only thing in which we can be truly confident is that no one – especially ourselves - sees the world clearly. But if we ignore this inconvenient truth during moments of important decision-making, we will likely come to regret it.

Fortunately, the steps to account for overconfidence bias in decision-making are fairly simple to take. The real challenge is getting people to actually take them. Why, especially for high-stakes decisions?

As with all cases of systemic irrationality, the answer lies in biases, heuristics and behavioral science.

The Brain's Limitations

The human brain has enough mental energy to process only a tiny fraction of the information available to it. Were people to try to think through every decision, they would become mentally depleted, and effectively paralyzed, by breakfast.

To compensate, human beings have adopted what Daniel Kahneman calls The Dual Systems Theory. Kahneman suggests that people engage in two very different types of thinking: System 1, which is fast, easy and unconscious, and System 2, which is slow, difficult and effortful.

By necessity, we spend the overwhelming amount of our time engaging in System 1 thinking (estimates typically reach 90% and above), reserving System 2 thinking for the relatively rare moments in the day in which we really need it.

If people are spending 90% of their time in unconscious autopilot, it begs another question: who's flying the plane? The answer to that is, "Biases and heuristics," the foundations of behavioral science.

Biases and Heuristics

Biases are mental tendencies, and heuristics are mental shortcuts that help us make decisions without expending mental energy. They fly the plane while we are in autopilot.

Fittingly, people have a bias against the word "bias," because they conflate it with racism, sexism, ageism, and all of the other isms that are worthy of our scorn.

Biases, however, are neither good nor bad. I have a positive bias for people who wear Canvas Chuck Taylor sneakers. As soon as I see them, I feel as though we have shared values and feel a kinship. It's just a mental tendency that helps me avoid spending any of my precious System 2 thinking.

Of course, bias can lead to all sorts of awful problems, especially when we perceive threats where there are none. But the answer is not to eliminate bias, because that is impossible. We could not survive without relying on biases. The challenge is learning how to recognize and mitigate the biases that our rational minds can understand are dangerous.

Heuristics face no such stigma, but play just as big a role in flying the plane in autopilot. Heuristics are mental shortcuts on which we rely when pressed to make decisions we can't "afford" to think through. The affect heuristic describes how we rely on emotions, or "our gut," to make decisions, rather than an extended thought process. Anchoring describes how we hold onto the first piece of information as our reference point in negotiations, without re-evaluating for accuracy or relevance. And the availability heuristic describes how ease of recall affects our perception of probability.

So, what is the cognitive purpose of overconfidence bias? The same as every other bias or heuristic: to save mental energy. On the plus side, imagine how much more exhausting life would be without it.

The Emotional Purpose of Overconfidence Bias

In modern culture, overcoming fear is considered an unequivocal good, a character builder and a character attribute. And to be fair, anyone who has overcome a fear knows how transformative an experience it can be.

But from an evolutionary psychology perspective, fear was wisdom. If only the strong survive, then fear is strength, because that's who survived. When you consider the preponderance of modern biases and heuristics tied to fear, (loss aversion, conformity, scarcity, stereotypes to name a few) it's clear that our ancestors emerged from an impossibly dangerous world by being afraid. Afraid of losing what they had, of expulsion from the herd, of running out, and of other people.

Anyone who has ever done bungee jumping, skydiving or the like knows that our conscious and unconscious minds can be in direct conflict. Zero-sum, winner-take-all conflict. Ironically, it is often the fear of being judged for quitting that beats out our fear of dying. But without overconfidence bias, no one would ever jump.

And that's as true metaphorically as it is literally. Overconfidence bias is necessary to get us to take action. We are wired for fear at a cellular level, so much so that competing biases had to emerge to ensure we didn't shrivel away in a cave from malnutrition.

But that is not the only role that overconfidence bias plays. It is often a necessary ingredient for people to achieve a measure self-esteem or find the motivation to improve themselves. Without some degree of overconfidence, we may not even believe in ourselves, in which case we will not try anything. And that is not an acceptable outcome.

Common Examples and Effects

Overconfidence bias affects our personal lives in countless ways:

And it has a similarly significant impact on our professional lives:

Our assessments of risks and rewards in decision-making are wildly influenced by overconfidence bias, so it is very much in our interest to understand how best to contain it.

Overconfidence Bias and Organizational Decision-Making

Biases don't exist in a vacuum. They interact with each other, affect one another, and in some cases are different symptoms of the same underlying cause. And when it comes to organizational decision-making, there is perhaps no deadlier combination of biases and heuristics than overconfidence bias and groupthink.

Groupthink describes how the capacities for individual analyses are affected by the presence of others. It encourages conformity.

Both overconfidence bias and groupthink encourage us to stop thinking and start doing. To be fair, in many contexts, that is exactly what is needed. But in many other contexts, it is fatal.

One of the more revealing findings from the work of Daniel Kahneman was tracking the performance of more than 8,000 financial traders over many years. What he found was zero relationship between performance from one year to another. In other words, no one was better than anyone else. They all had good years and bad years. The results were luck.

When Kahneman presented his findings to the executives, they weren't even surprised. They were able to take credit for their wins and attribute their losses to luck (fundamental attribution error) which cemented their status as "great." The saying "Often wrong, never in doubt" might as well have been written about the 8,000 traders he tracked.

But the problem doesn't end there. Organizational decisions are made by the people who ascended the ranks on the backs of their performance, be they substantive or political. That very path inevitably leads people to overestimate their judgment and foresight. Their advancement provides proof of their superiority, and as such their insights don't require the same level of skepticism as others'.

Research shows us that luck plays a far greater role in business success than skill, but try telling that to successful people. In most cases, that success becomes their identity, which means to them that they're usually, if not always, correct. (It doesn't actually mean that.)

Nonetheless decisions are being made by generally overconfident people subject to groupthink which discourages dissent. What's the worst that can happen?

The Existing "Solutions" for Overconfidence Bias

The solutions most commonly offered to combat overconfidence bias are very much aligned with most biases. Maintaining awareness during decision-making, using tools and processes to counter bias and heuristics, and relying on data to do so are among them.

And these solutions, when attempted, tend to produce great results. The problem is that these solutions are so infrequently attempted.

That is understandable for a number of reasons. For starters, that degree of evaluation is far too much work given the sheer number of decisions we have to make, and we have far too little mental energy to do so.

For individuals, better decision-making requires intention and discipline, while groups require coordination and facilitation. But even those not enough to overcome overconfidence bias, because they still fail to address the most powerful obstacle of all: emotional resistance.

The New Solution for Overconfidence Bias

If following a structured decision-making process might make someone's past or current decisions seem foolish, it eliminates their motivation to do it. No one wants to appear stupid, to themselves or especially to a group. If forced into a group structured decision-making exercise, they will be concerned exclusively with impression management, muddying the waters for an honest and objective evaluation.

The point must be made forcefully and effectively upfront: errors in judgment from biases and heuristics are as universal as breathing. Every single human has and will make countless of them, both consciously and unconsciously. And hindsight is always 20/20. To feel foolish for making errors in judgment is, quite literally, foolish. And it is a huge blocker to growth and improvement.

This mindset is a requirement for better decision-making, and facilitators must be skilled in making the point. But even that is not enough.

People's status and positions are often tied to certain decisions and/or supportive of the status quo. As such, they are strongly motivated to avoid or muddy any exercise that might threaten that order. This resistance must be eliminated as well. Fortunately, there's a framing to get us there.

Stories of agility success, like Play-Doh, Frisbee and Netflix, and of agility failure, like Kodak and Blockbuster, help to drive the point home.

What makes this the "new" solution to overconfidence bias is that it addresses the emotional blockers before proceeding to the cognitive exercises. Solutions are meaningless for people not motivated to use them.