What Are We Optimising For?

Summary:

• AI is automating reasoning, not just execution. Once reasoning is cheap, human judgment starts to look like friction.

• Some of that friction is waste. Some of it is how organisations create expertise, shared understanding, accountability and resilience.

• Competitive pressure rewards visible output more readily than these slower, harder-to-measure capabilities. The incentives need to change.

When judgment becomes the bottleneck

Software has always been in the business of removing friction: automating repetitive processes, eliminating handoffs, making systems faster and cheaper. AI extends that into cognition. Investigation, planning, comparison, implementation, testing and review can all potentially be handed to an agent.

An agent can investigate a problem, trace the cause, weigh the options, implement a solution and test it. Another agent can review the result. Given good context, the output can be remarkably strong, and when it fails the defects are often subtle rather than obvious.

Agents haven’t accelerated the SDLC evenly. They’ve compressed the time it takes to implement a change, while planning, review, validation and deciding what matters consume a larger share of the remaining human effort. The limiting resource is increasingly attention and judgment.

Engineers are also starting to act as proxies between machine-generated chains of reasoning. An engineer delegates an investigation to answer a colleague’s question; the colleague asks another agent to review the generated explanation. Neither person necessarily reconstructs the reasoning for themselves. Decisions can be produced, reviewed and approved without being fully understood by the people responsible for them.

That’s what is different about this wave of automation. We’re not only delegating execution; we’re increasingly delegating the reasoning and judgment that precede it.

Human judgment is slow by comparison. We hesitate, ask questions, disagree, gather more evidence, argue about trade-offs and sometimes realise we were asking the wrong question. To a system optimising for throughput, all of that looks like latency. Once machine reasoning becomes cheap, human reasoning stops looking like the work and starts looking like friction in the way of the work.

Some of that friction is waste. But some of it was quietly doing other jobs: developing expertise, spreading understanding across a team, making someone genuinely answerable for a decision, and keeping the organisation able to cope when things go wrong in ways nobody predicted.

So the real question isn’t how much friction we can remove. It’s which friction was carrying things we never thought to measure.

“Isn’t this just another abstraction layer?”

Every generation of programmers has watched part of its craft disappear. Assembly programmers worried that compilers would produce engineers who didn’t understand the machine. They were right, and it mostly didn’t matter. Frameworks, cloud platforms and Stack Overflow each let people build more while understanding less of what sat underneath.

Traditional abstractions remove mechanical execution. A compiler decides how my program runs on the hardware, but the why stays with me: what the program should do, how the problem is modelled, which trade-offs matter. It also operates within a narrow, well-understood contract. The range of things it’s allowed to decide is small.

We’ve been here before: the ironies of automation

In 1983 the psychologist Lisanne Bainbridge published a four-page paper called Ironies of Automation. Her observation was that automating the routine parts of a job doesn’t simplify the human role; it concentrates it. People are left with the rare, ambiguous and novel situations the automation can’t handle, while losing the everyday practice that builds the knowledge those situations demand.

As she put it:

“By taking away the easy parts of his task, automation can make the difficult parts of the human operator’s task more difficult.”

Another line reads almost like a forecast. Automated systems monitored by former manual operators were “riding on their skills, which later generations of operators cannot be expected to have.”

Aviation became the textbook case. Automation made flying dramatically safer, and pilots spent less time flying by hand. In 2013, after flight data showed an increase in manual-handling errors, the FAA issued a safety alert encouraging airlines to promote manual flying when appropriate. Automation wasn’t the mistake, but the skills it eroded didn’t maintain themselves, so the industry deliberately put some friction back.

Bainbridge saw that coming too: “it is the most successful automated systems, with rare need for manual intervention, which may need the greatest investment in human operator training.”

Where do senior engineers come from?

Senior engineers aren’t hired as senior engineers. They’re shaped, slowly, by ordinary work: writing code that turns out to be wrong, misreading systems, debugging a production issue, making design decisions that age badly, taking review comments that sting, sitting through incident reviews.

The optimistic take is that AI handles the routine work so humans can focus on the hard decisions. But the ability to make hard decisions is built out of years of routine ones.

We increasingly need human expertise to supervise AI, while automating the activities through which humans acquire that expertise.

There are early signs this is more than a thought experiment. Stanford’s Digital Economy Lab, using payroll data covering millions of US workers, finds no evidence of widespread AI-driven job losses. But among 22–25-year-olds in the most AI-exposed occupations, employment is now about 19% below where it would have been had it tracked similar workers in less-exposed jobs. Experienced workers show no comparable gap. The difference comes mainly from reduced hiring, not layoffs.

The latest revision adds a telling detail. Employment has fallen for young workers in jobs built on codified knowledge — the textbook kind — while rising for experienced workers in jobs that depend on tacit knowledge “acquired through practice, mentorship, and repeated exposure to real situations.”

The researchers are careful to call this descriptive rather than causal, and so should we. But it’s exactly the shape of signal the paradox predicts: employers still hiring for experience while hiring fewer of the people who would go on to gain it.

If we remove the bottom rungs of the ladder, we shouldn’t assume people will keep appearing at the top.

More output isn’t more capability

Education makes the distinction easy to see. With AI, a student can produce work far beyond what they could manage alone. Their performance has improved; whether they’ve learned anything is a separate question.

The OECD’s Digital Education Outlook 2026 finds that general-purpose AI can raise the quality of students’ work without matching learning gains. In several studies, the advantage disappears or even reverses once the AI is taken away. In one experiment with nearly a thousand high-school maths students, those who had used an unrestricted GPT-4 tutor scored 17% lower on the exam than students who never had it — and didn’t seem to realise their learning had suffered.

Hartzog and Silbey make the institutional version of the same argument in How AI Destroys Institutions. Their concern is that AI doesn’t just risk weakening individual expertise; at scale it can erode the human processes through which institutions develop and maintain it. An institution can become more productive while getting worse at producing the human capability it will later depend on.

Knowledge work shows a similar pattern. Microsoft Research surveyed 319 knowledge workers about 936 real uses of generative AI. Greater confidence in the AI was associated with less reported critical thinking, while effort shifted from solving problems towards verifying and integrating AI output.

In software, a small randomised trial from Anthropic had 52 mostly junior developers learn an unfamiliar Python library, with or without AI help. The AI group wasn’t significantly faster, but scored 50% on a comprehension quiz against 67% for those who worked by hand. The biggest gap was in debugging: precisely the skill you need to notice when generated code is wrong. How people used the AI mattered too; those who asked conceptual questions did well, while those who simply delegated did not.

AI doesn’t make people worse. But output and capability are different quantities, and AI can move the first without moving the second. If I ship 5x as much software with agents, I have 5x the output. That doesn’t make me 5x as capable. Orchestrating agents well is a skill, but it’s not the same thing as understanding five times as much.

The danger is that the capability most vulnerable to erosion is also the one verification depends on.

How humans drift out of the loop

Now scale this up to the “AI-native” organisation. Most AI maturity models end with AI embedded in the operating model and human checkpoints managing the risk, but I suspect the progression doesn’t stop there:

1. Humans work; AI assists.

2. Humans decide; AI executes.

3. Humans specify outcomes; AI plans and executes.

4. AI plans, executes and verifies; humans approve.

5. AI handles routine approvals; humans handle exceptions.

6. AI resolves increasingly complex exceptions; humans set objectives and constraints.

None of those steps looks obviously reckless on its own. Generation gets cheap, so review becomes the bottleneck and we automate first-pass review. Then approval becomes the bottleneck, so low-risk changes get auto-approved. Eventually exceptions dominate, so we automate the common ones too.

Each change is local, reasonable and backed by a metric. Follow that gradient far enough and the human moves from the centre of the loop to its edge without any single decision that looks like the moment it happened. Simply keeping a “human in the loop” doesn’t necessarily fix it either.

During the 2026 war with Iran, an intelligence report circulated through the US military claiming a Chinese vessel was carrying components of a nuclear weapons programme. According to reporting on the incident, aircraft were already in the air and armed personnel were preparing to board the ship when officials examined the intelligence more closely. The report was false: an analyst had used an AI chatbot that misidentified the cargo, then used AI again to turn the result into a formal intelligence report.

Human oversight eventually caught the error, but very late in the chain. Humans can remain formally in the loop while becoming increasingly dependent on machine-generated reasoning they haven’t independently verified.

At the other end of the spectrum, startup C5R’s Facility-0 is a wet lab where frontier models can take a scientific goal, design experiments as code, direct instruments and human operators, analyse the results and decide what to try next. The people in the lab can, quite literally, become the hands of the machine. This is why autonomy is not the same thing as maturity.

Picture an organisation with thousands of agents that can understand, decide, act and verify with little human involvement. On most maturity models it looks incredibly advanced. Now suppose nobody inside it can explain an important decision without asking the system that made it. It struggles with novel failures, no longer develops experienced people because much of the work they would have learned from has disappeared, and its approvals come from people who can’t meaningfully interrogate what they’re approving.

That organisation may be extremely capable and cognitively hollow: impressive during normal operation, brittle exactly when it matters most. At an organisational scale, comprehension debt becomes epistemic debt:

The accumulated difference between what an organisation’s systems can do and what its people understand about why.

Why “just be sensible” won’t work

Even if you agree with all of that, “just be sensible about AI” isn’t much of a strategy because the incentives point the other way.

Goodhart’s law says that when a measure becomes a target, it ceases to be a good measure. Throughput, cycle time, cost per change and degree of automation are all legitimate things to measure, and they’re visible almost immediately. Expertise, shared understanding, accountability and resilience are slower to show up, harder to attribute and much harder to put on a dashboard, so they can quietly disappear from the objective function.

AI introduces a closely related problem: specification gaming. Give an optimiser a measurable proxy for what you really want and it can become extremely good at satisfying the proxy without satisfying the intent. It’s Goodhart’s law with an optimiser attached.

As organisations delegate more decisions to agents, those two effects start to reinforce one another. The organisation optimises what its dashboards can measure; its agents optimise what the organisation can specify. Maintainability, understanding, judgment and resilience are vulnerable at both levels precisely because they rely so heavily on context and human interpretation.

And firms don’t make these choices in isolation; they compete. Imagine two companies that both care about deep engineering expertise. One protects time for engineers to investigate problems, mentor juniors and occasionally do work an agent could have done faster. The other automates everything it technically can. If the second company gets similar results at lower cost, the first is under pressure to follow.

The cost of preserving capability is visible now, while the benefit might show up during a rare production failure, five years later in the judgment of an experienced engineer, or in a senior engineer who works for somebody else by then. The company training junior engineers pays the cost, but part of the return goes to their future employers and to the profession more broadly. The company maintaining redundant human understanding pays for it every day, but gets most of the payoff only when something unusual goes wrong.

Markets aren’t guaranteed to produce the right amount of either. Bainbridge’s irony comes back around: the more successful the automation, the more investment in human capability it may require — and the harder that investment becomes to justify.

We’ve seen this pattern before

Short-term efficiency eating away at long-term capability isn’t new. In a well-known survey of 401 financial executives, around 80% said they would cut discretionary spending on R&D, advertising or maintenance to hit an earnings target.

Manufacturing gives us an even closer parallel. In 2009, Harvard’s Gary Pisano and Willy Shih argued in Restoring American Competitiveness that decades of outsourcing, sensible company by company, had eroded America’s “industrial commons”: the shared know-how, engineering skill and manufacturing competence that future innovation depends on.

Their argument was that today’s apparently low-value manufacturing work can contain the seeds of tomorrow’s innovation. Outsource enough of it and the capacity to innovate goes with it. The mechanism here isn’t identical, but the shape is familiar: a locally rational decision, an immediate measurable gain, and a delayed cost shared by everyone. Swap “low-value manufacturing work” for “junior engineering work” and the analogy becomes uncomfortable.

A profession’s expertise pipeline behaves a lot like an industrial commons, and AI makes it cheaper than ever to stop contributing to it.

Scott Alexander and Daniel Schmachtenberger call this kind of structure a multipolar trap: individually rational actors pushed towards a result that none of them would choose collectively. The same structure may apply one level higher, to the development of AI itself. In Racing to the Precipice, Stuart Armstrong, Nick Bostrom and Carl Shulman modelled a race to build transformative AI. If being first carries enough advantage, each competitor has an incentive to move faster and accept more risk, even if all of them would prefer a safer race.

It’s only a model, not a prediction, but the shape of the problem is the same: what is rational for each participant can be worse for everyone. You can see the same coordination problem at different scales of human organisation: firms under-investing in their future expertise, a profession hollowing out its training pipeline, or AI developers racing towards capabilities none of them would necessarily choose to pursue under the same conditions collectively.

When wisdom is competitively expensive, asking individual firms to be wiser isn’t a strategy.

So what are we optimising for?

The usual question about AI is can this be automated? As capabilities improve, the answer will increasingly be yes.

I think the more useful question is what are we giving up if we do?

Sometimes the answer really is nothing. It’s waste, and we should automate it without hesitation. But other times, what looked like inefficiency was doing another, often invisible, job.

Yes, this post was generated with AI and edited by a human. It seemed fitting.

--yolo