Why Smart Systems Make Stupid Decisions

by Loring Mortensen

In the years leading up to the 2008 financial crisis, the global financial system was staffed by some of the most credentialed and analytically capable people on the planet. The major banks employed mathematicians, physicists, and economists with degrees from the world's best universities. Risk models were sophisticated. Regulatory bodies had access to the same data the banks did. Warnings about the housing market appeared in major publications years before the collapse. By any measure of organizational intelligence, the system was functioning at an extraordinarily high level.

And yet it produced an outcome that almost no one inside it would have chosen.

The standard explanations have been argued about ever since. Greed. Deregulation. Political capture. Bad incentives. Faulty models. Each explanation contains some truth, but none of them really accounts for what is most strange about the event — that thousands of intelligent people, looking at the same data, working in their own narrow areas of competence, each making decisions that were locally defensible, collectively produced a catastrophe that destroyed enormous wealth and then required public intervention on a scale that none of them would have predicted.

This is not unique to finance. The same shape appears elsewhere.

Climate change has been understood scientifically for decades. The data is uncontested across the relevant fields. Governments, corporations, and international bodies have invested enormous intelligence in studying it, modeling it, and proposing responses. And yet the gap between what the system knows and what the system does has remained stubbornly wide, in ways that cannot be explained by ignorance or by the failures of any single actor.

Persistent inequality follows a similar pattern. The economic data is available. The historical analysis is rich. Policy proposals exist across the political spectrum. Smart people in many domains have studied it for generations. And yet the structures that produce it continue to operate, often with the active participation of people who would describe themselves as opposed to the outcomes they help produce.

These are not failures of intelligence. They are something else.

The standard frame assumes that better thinking would produce better results — that if only the analysis were sharper, the data cleaner, the experts more numerous, the outcomes would improve. But the financial crisis was not caused by a shortage of analysis. Climate inaction is not caused by a shortage of climate science. Inequality is not caused by a shortage of economic understanding. In each case, the intelligence is there. The intelligence has been there for a long time. Something else is missing.

The thing that is missing is coherence.

Intelligence is the capacity to process information, generate options, and identify good answers within a defined problem. Systems are very good at intelligence. They have committees, analysts, models, and dashboards for evaluating almost anything that can be specified clearly. When the question is well-formed, these systems usually produce reasonable answers.

Coherence is something different. Coherence is the property of a system whose parts remain aligned with the purpose the whole was meant to serve. It is what allows good answers to add up to good outcomes rather than canceling each other out. It is what keeps the activity at the surface connected to the meaning underneath it.

Intelligence happens locally. Coherence is a property of the whole system over time.

A smart system can lose coherence without losing intelligence. Each part can be doing excellent work. Each decision can be defensible on its own terms. Each metric can be improving. And yet the sum of all this activity can be moving in a direction that no one inside the system actually wants. The intelligence is real. The integration is missing.

This is hard to address because no part of the system is responsible for coherence. Departments are responsible for their domains. Experts are responsible for their fields. Officials are responsible for their portfolios. Each role is bounded. But the coherence of the whole — the question of whether all this excellent work actually adds up to what the system was meant to do — has no owner. It is everyone's concern and no one's job.

What makes this worse is that coherence is invisible when it is working. When the parts of a system are aligned with the whole, the system simply functions, and no one notices the alignment because nothing is going wrong. Coherence becomes visible only when it is lost — and by then, the loss has usually been accumulating for a long time, distributed across thousands of small decisions that each looked reasonable when they were made.

The result is what we are now living through across multiple domains at once: intelligent systems producing outcomes their own participants find inexplicable. Smart people, smart institutions, smart processes — and decisions that no one would have chosen if they had been able to see the whole picture.

The question worth sitting with is not why these systems are failing to be smart. They are very smart. The question is what they are missing that all the intelligence in the world cannot supply.

"When a complex system is far from equilibrium, small islands of coherence in a sea of chaos have the capacity to shift the entire system to a higher order."

— Often attributed to Ilya Prigogine — a paraphrase, perhaps, of his work on dissipative structures and self-organization in systems far from equilibrium.