An Unsophisticated Decision-Making Machine
Rubber-stamping an AI's "opinion" is not the same thing as making a decision
The futurist Gerd Leonhard has the best analogy for the possibilities and dangers of AI I know. His framing: AI is a power tool. A carpenter with a hammer sets nails one at a time. A carpenter with a nail gun works twenty times faster. People with the power tool beat people without it — that’s just true, and pretending otherwise isn’t principled, it’s just slow. But, he says, you wouldn’t hand the nail gun to your kid to play with. You have to understand what the thing does. You have to keep asking questions. And above all, don’t let AI make “decisions that we should have made” just because we couldn’t be bothered.
That’s right, and it’s generous. It’s also, I think, the comfortable version of the problem because it lets you off the hook. I’m not lazy, you think. I read carefully. I check things. That warning isn’t about me.
I’d like to show you why it might be. I know I’m still constantly reminding myself.
An opinion is not a decision
Here’s the thing about these machines that took me an embarrassingly long time to understand.
An AI always has an opinion. It cannot not have one. Ask it anything, anything at all, including things no one could sensibly answer, and it will produce a position, because producing positions is the only move it has. There’s no version where it shrugs.
And a machine that always has an opinion looks an awful lot like a machine that makes decisions.
It isn’t one. A real decision has parts, and the machine can manufacture all of them but one.
It can produce the choice — this option, not that one. It can produce the reasons — here’s why, in three tidy points. It can produce the next step — and here’s what to do Monday morning. Those three things, arriving together in confident, well-organized prose, are exactly what a decision looks like from the outside.
What it cannot produce is the fourth part: somebody who has to live with it.
No stakes. No skin in the game. No lying awake at two in the morning wondering if you got it wrong. The philosopher Martha Nussbaum argued that our emotions aren’t just fuel for thinking — they’re part of the thinking itself, tangled up in how we weigh what matters. The machine has none of that. It has no desire pushing it toward the answer and no feelings about the answer afterward. It will never once care what happens to you.
It can produce every part of a decision except the part that makes it one: somebody who has to live with it.
So what arrives in your chat window is not a decision. It’s a suggestion wearing a decision’s clothes. And the better these machines get, the better the costume fits.
The pilots who shut down a healthy engine
Now let me tell you why “just don’t be lazy” isn’t enough.
This problem is older than chatbots. Researchers who study how humans work with machines have a name for it — automation bias — and the best-known studies are from the 1990s, done with flight simulators and, crucially, with professional airline pilots.
Linda Skitka and her colleagues ran an experiment. Pilots flew a simulated flight with a computerized aid that monitored the plane’s systems and made recommendations. The aid was very reliable, but not *perfectly* reliable.
At one point, the system announced an engine fire. There was no fire. Nothing else in the cockpit — none of the other gauges and indicators, all of which were completely accurate — supported the alarm.
Virtually every pilot shut down the engine.
A hundred percent of them, in that condition. Then, on the questionnaire afterward, those same pilots said that a single warning message with no other supporting signs would not be enough to diagnose a fire, and that shutting down an engine on that basis would be unsafe.
They knew the rule. They could state the rule. They followed the machine anyway.
And there’s a second finding that I think about constantly. When the automation failed to warn them about a real problem, pilots missed it about half the time — and how likely they were to miss it correlated with their total flight hours and years of experience. The more experienced the pilot, the worse they did at catching what the machine missed.
Not the careless ones. Not the lazy ones. The seasoned ones — the people who’d trusted good systems for so long that trusting a new one had become automatic.
This is not a story about sloth. It’s a story about what confident, competent-sounding output does to human judgment. And AI chatbots are the most confident, most competent-sounding advice machines ever built. The prose is clean. The logic is legible. Nothing on the surface says this is a guess assembled from patterns in text.
So you don’t lazily accept it. You read it. You nod. You think yes, that’s what I would have concluded and you agree.
It felt like judgment, but it was simply recognition.
What you become when you don’t decide
There’s a second cost, and this one is the most worrying.
The anthropologist Madeleine Clare Elish studied a string of accidents involving highly automated systems — planes, nuclear plants — and noticed a pattern in who got blamed. She named it the moral crumple zone.
In a car, the crumple zone is the part designed to absorb the force of a crash so the driver survives. In a complicated automated system, she argues, the nearest human being becomes that part — the component that absorbs the moral and legal impact when the whole thing fails. Even when that person had very little real control over what happened. The crumple zone ends up protecting the system’s reputation, at the expense of the human standing closest to it.
You can end up there without ever having made a decision.
If you approved the AI’s answer rather than deciding with it — if you were the human in the loop in name only, the signature at the bottom, the one who said “looks good” — then when it goes wrong, you are the only party in the entire arrangement who can be held responsible. The machine has no stake. The company that made it has terms of service. You have your name on it.
Which means the question isn’t only did I check the work? It’s did I actually decide, or did I just agree?
Four habits for better human decision making
None of this is an argument against using AI. I use it every day, for real work, and it has made me faster and in some ways better. The nail gun is genuinely better than the hammer.
But there’s a difference between using a tool and handing it your judgment. Four habits keep them separate:
Decide who decides, before the work starts, not after. This one has evidence behind it. In a follow-up study, the same researchers found that telling people in advance they’d be accountable for the accuracy of their decision measurably reduced automation bias. Knowing beforehand that it’s yours changes how you engage. Finding out afterward isn’t accountability it’s just blame.
Ask for options, not an answer. This is possibly the easiest yet most important habit to adopt when working with AI. One answer invites agreement. Two or three options with honest trade-offs force you to actually weigh them, which is the real work. If the AI hands you a single path, ask what it ruled out and why.
Write down why, not just what. If your only reason is “the AI suggested it and it sounded right,” you haven’t decided. You’ve delegated, and kept the liability. A decision you can’t explain is one you can’t defend, teach, or repeat.
Ask yourself what would change your mind. If the honest answer is nothing, you weren’t deciding. You were confirming. It’s the cheapest test I know and it takes about eight seconds.
Back to the carpenter
Leonhard’s carpenter is right where it counts. Refusing the nail gun out of nostalgia isn’t a virtue, and the people who learn to use it well will run rings around the people who won’t touch it.
But you don’t hand the nail gun to your kid — and you don’t hand it your judgment, either. The tool has no view about where the wall should go. It has never once cared whether the house stands.
Your AI always has an opinion. It has never, not once, had a stake in what happens to you. Don’t confuse the two, especially not on the days when its opinion is better than yours. That will happen. That’s exactly when the AI decision making costume fits best.
The decision is always yours.
The only question is whether you notice making it.
I teach this thinking and how to build a working agreement with an AI that keeps decisions where they belong in a six-week course called Designing Trustworthy AI Experiences. If that’s your kind of thing, come along.