MH
← All essays
Originally on Substack

What Are We Becoming?

On artificial intelligence, human agency, and the hidden costs of progress.

Like a lot of people, I’ve become increasingly wary and exhausted by generative AI. Some of what I’m seeing genuinely unsettles and alarms me. I’ve tried to put those thoughts into a series of essays (linked at the end). This post is my attempt to bring them together.

Over the past year, AI has increasingly hollowed out the work I do as a software engineer. It has alienated me from my job and taken much of the meaning and satisfaction I used to find in it. It has pushed my decision-making from the centre to the periphery. Many engineers, including me, now find ourselves acting as glorified button-pushers. Or, more bleakly, “meat proxies”.

This sense of foreboding has led me somewhere I didn’t expect. I started out worrying about how AI is affecting my capacity to do my job well, and somehow I ended up thinking about existential risk.

The risk I mean isn’t dramatic, with Skynet or Terminators. It’s subtler and more mundane: a withering away of human agency in the world. We don’t get overthrown. We get optimised out of the loop, one reasonable decision at a time, until we’re no longer needed for the things that used to depend on us. A group of researchers led by Jan Kulveit calls this gradual disempowerment. They argue it could amount to an existential catastrophe: not necessarily extinction, but the permanent loss of humanity’s influence over its own future. Alongside that sits a philosophy called AI successionism, the idea that AI should eventually inherit the future. My worry is that successionist views could end up justifying that outcome as fine.

In my first essay, I looked at how AI is eroding our ability to understand the systems and processes we depend on. Some call this comprehension debt: the growing gap between what a system can do and what we understand about why it works. Scale that up, and it becomes something more like collective epistemic debt. It isn’t only software engineers who are experiencing this gap. METR, an independent organisation that evaluates AI, has warned that as AI increasingly automates AI research itself, progress could outpace researchers’ ability to understand and control it. On 17 October, Jack Clark, a co-founder of Anthropic, told the IMF’s annual meetings that he believes the singularity has now begun. He said we’re already in a world where AI makes discoveries that humans then have to spend time decoding after the fact.

To put it plainly, I believe we’ve already lost control of AI. In a very real sense, nobody is at the wheel, or at best they are drunk or half-asleep, or deluded by the hubris of their own grandiose vision of the future, which they want to impose on the rest of us. What we’ve lost is collective control over where all this is going. We can still make decisions about individual systems, but we don’t seem able to make decisions about the whole. No lab can easily slow down without risking being overtaken. No government has yet shown it can set the pace. And the people building these systems increasingly rely on the systems themselves to understand what they’ve built. These dynamics reinforce one another. So while METR talks about loss of control as a future possibility, I see a form of it as a present reality, and part of the larger meta-crisis.

If the incentives stay as they are, if our institutions don’t adapt, and if we keep optimising for whatever is easiest to measure, then gradual disempowerment, and perhaps the philosophy that justifies it, starts to look like the default outcome. Not because anyone chose it, but because nothing else changed.

So let me start where this began for me: software engineering. That’s where I can see these changes up close. From there, I want to widen the lens to institutions, and ask whether we’d even notice if we were losing our ability to shape the future. Then I want to get to the question I find hardest. Even if we got everything right, even if we kept control and shared the benefits, would the life that remained be worth living?

Before any of that, something small. Imagine you’ve hurt someone and you need to write them an apology, but you can’t find the words. I’ll come back to that letter at the end, because I think it holds the whole argument.

I don’t pretend to have all of this figured out. Some of what follows is supported by solid evidence, and some of it is a hypothesis I find compelling. I’ll try to say which is which.

Where I started: software engineering

Software engineering has always been about removing friction. We automate repetitive processes, eliminate handoffs and make systems faster and cheaper. AI extends that into cognition itself. Investigation, planning, implementation, testing and review can potentially all be handed to an agent. Give an agent clear enough context and it can trace a problem to its cause, weigh up options, implement a fix and test it. When it fails, the mistakes tend to be subtle rather than obvious.

What’s different this time is that we’re not just delegating execution. We’re increasingly delegating the reasoning and judgement that come before it. Agents have compressed the time it takes to implement a change, so planning, reviewing and deciding what matters now take up a bigger share of what’s left. Attention and judgement have become the limiting resources.

Increasingly, software engineers also act as go-betweens for chains of machine-generated reasoning. I might ask an agent to investigate a colleague’s question. My colleague might ask another agent to review the explanation. Neither of us has necessarily worked through that reasoning ourselves. A decision gets produced, reviewed and approved without ever being fully understood by the people responsible for it. That’s what I mean by a “meat proxy”.

The trouble is that human judgement is slow. We hesitate. We ask questions. We disagree, gather more evidence, and sometimes realise we’ve been asking the wrong question altogether. To a system optimised for throughput, all of that looks like latency. Once machine reasoning becomes cheap, human reasoning stops looking like the work and starts looking like friction in the way of the work.

Some of that friction is waste. But some of it was quietly doing other jobs. It was building expertise, spreading understanding across a team, making people answerable for decisions, and keeping an organisation capable of coping when things go wrong in ways nobody predicted. So the question isn’t simply how much friction we can remove. It’s which friction was carrying things we never thought to measure.

For me, AI should remain a tool: a means to an end that serves people, without taking away their autonomy or agency. It shouldn’t become an end in itself, and it shouldn’t replace the person using it. A good tool extends what you can do while leaving you better able to understand, decide and act. If it does the opposite, we need to ask what we’re actually gaining by using it.

It’s tempting to say we’ve been here before. Assembly programmers worried that compilers would produce engineers who didn’t understand the machine. In a sense they were right, but mostly it didn’t matter. A compiler decides how my program runs. The why still belongs to me, and the compiler works within a narrow, well-understood contract. This time, we’re beginning to hand over the why as well.

Part of this problem has a name, and it’s older than AI. In 1983, the psychologist Lisanne Bainbridge wrote a short paper called Ironies of Automation. Her observation was that automating the routine parts of a job doesn’t simplify the human role. It concentrates it. People are left with rare, ambiguous, unfamiliar situations the automation can’t handle, while losing the everyday practice that would have prepared them for exactly those situations.

Ask a simple question: where do tomorrow’s senior engineers come from? Nobody starts out as a senior engineer. People become one slowly, through ordinary work. They write code that turns out to be wrong. They misunderstand a system. They debug production failures. They make design decisions that age badly, and they take review comments that sting a little. The optimistic story is that AI takes away the routine work so humans can focus on the hard decisions. But our ability to make those hard decisions is built from years of making the routine ones. We’re increasingly asking human experts to supervise AI, while automating the very activities through which people become experts.

Imagine a graduate called Dave. Dave is a composite, not a real person, but the pattern is real. He finished a computer science degree last year and started applying for junior roles, the jobs that used to be the way in. Many companies now use agents for much of the routine work those roles once covered. The work Dave would have learned from is finished before he gets near it. Nobody decided Dave shouldn’t get a chance. Each manager made what looked like a sensible decision.

And the early numbers look a lot like Dave’s story. Stanford’s Digital Economy Lab analysed payroll data covering millions of US workers. They found no evidence of widespread job displacement across the whole economy. But among 22-to-25-year-olds in occupations exposed to AI, employment was about 19 percent below where it would have been had it kept pace with less exposed occupations. Experienced workers showed no comparable gap, and the difference appeared mainly through reduced hiring rather than layoffs. The researchers are careful to say this is descriptive, not proof that AI caused it, and we should be just as careful. But it’s exactly the kind of signal this paradox would lead us to expect. If we remove the bottom rungs of the ladder, we shouldn’t assume people will keep appearing at the top.

The clearest example I’ve found is in medicine. Researchers in Poland studied experienced endoscopists at four centres that had started using AI to help detect polyps during colonoscopies. Each of these doctors had performed more than two thousand procedures. When they later carried out colonoscopies without AI, their detection rate for precancerous growths had fallen from 28.4 percent to 22.4 percent (the Lancet study). The study is observational, so it can’t prove AI caused that decline. The number of cancers detected didn’t fall either, with six cases before and eight after, although those numbers are very small. Still, the warning is worth taking seriously. Regular reliance on a tool may erode the skill it was meant to support, and you may only discover that when the tool is no longer there.

Another trap is confusing output with capability. With AI, I can produce far more than I could alone. My output has improved. Whether I’ve actually learned anything is a separate question.

Anthropic ran a randomised trial on exactly this. Fifty-two mostly junior developers were asked to learn an unfamiliar Python library, some with AI assistance and some without. The AI-assisted group wasn’t significantly faster, but they scored 50 percent on a comprehension quiz, compared with 67 percent for those who worked by hand. The biggest gap was in debugging, which is precisely the skill you need when generated code goes wrong. How people used the AI mattered too. Those who asked conceptual questions did reasonably well. Those who simply delegated the work did badly. The study is by Anthropic, the company behind Claude, so weigh its origin accordingly. Microsoft researchers found a related pattern in a survey of 319 knowledge workers. The more they trusted AI, the less critical thinking they reported doing. That’s self-reported, so it’s a signal rather than a direct measurement.

None of that proves AI makes people worse at their jobs. It shows that output and capability are different things. If I ship five times as much software with agents, my output has increased fivefold, but that doesn’t make me five times as capable. The skill most at risk of erosion may be the very skill I need to check whether the AI has done a good job.

Now imagine that pattern spreading through an organisation. Humans drift out of the loop without anyone making an obviously reckless decision. Generating software becomes cheap, so reviewing it becomes the bottleneck. We automate the first pass of review. Then approvals become the bottleneck, so we automatically approve low-risk changes. Eventually, handling exceptions is what slows us down, so we automate the common exceptions too. Each change is local, reasonable and supported by a metric. Follow that path far enough, and people move from the centre of the loop to its edges, without any single moment when someone decided to remove them.

Simply keeping a human “in the loop” doesn’t necessarily solve it. During the war with Iran this spring, according to CNN’s reporting (carried by TechCrunch), an analyst at US Special Operations Command Pacific used an AI chatbot to analyse intelligence about a Chinese ship. The chatbot concluded the ship was carrying components of a nuclear weapons programme. The analyst then used AI again to turn that into a formal intelligence report, which circulated through the military. Aircraft were reportedly already in the air, and armed personnel were preparing to board, when officials looked more closely and realised the claim was false. Human oversight caught the error, but only very late in the process.

Picture an organisation where thousands of agents can understand, decide, act and verify, with almost no human involvement. By most maturity models, that looks incredibly advanced. But suppose nobody in it can explain an important decision without asking the system that made it. Suppose it struggles with unfamiliar failures, and no longer develops experienced people, because the work they would have learned from has disappeared. That organisation might be extraordinarily capable and, at the same time, cognitively hollow: impressive when everything runs normally, brittle when it really matters. At that scale, comprehension debt becomes epistemic debt: the accumulated gap between what an organisation’s systems can do and what its people understand about why.

You might be thinking, “Well, we’ll just be sensible about it.” I wish it were that easy. The incentives point the other way. Throughput, cycle time and cost per change show up almost immediately. Expertise, shared understanding, accountability and resilience take longer to appear, and they’re much harder to put on a dashboard. This is Goodhart’s law: when a measure becomes a target, it stops being a good measure. AI adds a twist. Give an optimiser a measurable proxy for what you really want, and it can become very good at maximising the proxy without serving the purpose behind it.

Companies don’t make these decisions in isolation. Imagine two firms that both care about deep engineering expertise. One protects time for engineers to investigate problems, mentor juniors and occasionally do work an agent could do faster. The other automates everything it can. If the second delivers similar results at lower cost, the first is under pressure to follow. The cost of preserving expertise shows up now. The benefit might only appear during a rare production failure, or in the skills of a future senior engineer who may be working somewhere else by then.

We’ve seen a version of this before. Decades of perfectly sensible decisions, made company by company, can hollow out the shared know-how of an entire industry, the very knowledge future innovation depends on. Professional expertise works the same way, and AI makes it cheaper than ever to stop investing in it. Each firm does the sensible thing for itself, and together they lose something none of them would have wanted to lose. When wisdom is competitively expensive, asking individual firms to be wiser isn’t really a strategy.

And this is where my unease stops being just about my job. That same pattern doesn’t end at the office door. It can play out across professions, governments and, ultimately, the race to build AI itself.

Beyond software: can we still notice?

So let’s widen the lens. Everything I’ve described so far comes from software engineering, but the underlying logic isn’t unique to software. Kulveit and his colleagues explore it in their 2025 paper on gradual disempowerment. Think about why powerful institutions pay attention to ordinary people at all. Businesses need workers and customers. Governments need citizens, taxes and consent. Culture needs people to create it and people to enjoy it. These relationships are far from perfect, and certainly not always fair, but dependence gives institutions a practical reason to care what people want.

As AI starts doing more of that work, the dependencies loosen, and with them the reasons to listen. Nobody has to intend that outcome. A company cuts costs. A government agency automates something frustrating. A school adopts a new tool. Each choice seems reasonable on its own. Paul Christiano sketched a similar possibility back in 2019: not an explosion, but a slow fade.

A more troubling question follows. If that slow fade were already under way, would we notice?

A writer called Karim Abdul-Aal, writing on the Effective Altruism Forum, raises a difficult question about these models. They often seem to assume a healthy starting point: that society is broadly well aligned, that people can think clearly, and that institutions can correct course. But what if the erosion began before AI? What if weakened institutions and a declining ability to reason together aren’t simply consequences of AI dependence, but the conditions in which it takes hold? Abdul-Aal is clear that this is a synthesis, not a proof. So I’ll approach it with the same caution, and try to separate stronger evidence from weaker.

Education

The latest Education Scorecard, from researchers at Stanford, Harvard and Dartmouth, finds that American maths and reading scores have been declining since around 2013, well before COVID. The researchers point to two possible contributors: the fading of test-based accountability, and the rise of social media. But they’re careful not to claim either has been proved to cause the decline. There’s a slow recovery under way, although no state has returned to its 2013 level.

Among adults, a large international survey called PIAAC found that the share of Americans at the lowest literacy level rose from 19 percent to 28 percent between 2017 and 2023, according to the NCES release. Literacy isn’t the same as a society’s capacity to think together. But it’s measurable, and so much of our ability to read, weigh arguments and make decisions depends on it.

Media

The evidence here is more contested. One familiar story is that the repeal of the Fairness Doctrine in 1987 helped open the door to openly partisan talk radio, and that media consolidation later pushed news towards capturing attention. The first part is plausible, but AM radio was also losing listeners to FM at the time, and stations were looking for new formats, so it’s hard to separate those influences. The second part is less clear-cut still. A recent study of local television consolidation, cited in Abdul-Aal’s post, found major changes to news content, but the direction depended on the owner, and viewers barely reacted. I wouldn’t claim we have a simple, settled explanation.

Governance

A well-known 2014 study by Martin Gilens and Benjamin Page examined around 1,800 policy issues over two decades. It found that when ordinary citizens disagreed with economic elites and organised interest groups, the citizens generally lost. It’s influential, and heavily debated. Critics point out that the preferences of average and wealthy citizens often overlap, which makes their separate influence hard to pin down. So I’d treat it as a serious finding, not the last word.

Put those three areas together. Education affects how we think. Media affects what we think about. Governance affects what we can actually do about it. If each drifts towards its own narrow targets, they can reinforce one another, and the early-warning system weakens exactly when we most need it. The researcher Subramanyam Sahoo has suggested that disempowerment may even be self-concealing: a culture that has drifted away from human flourishing might not be recognised as such by the people living inside it.

So what do I take from all of that? It doesn’t prove that our institutions are already hollow. Much of the evidence is correlational, some of it is contested, and some trends are improving. But Abdul-Aal’s narrower point is hard to dismiss: the starting conditions for governing AI may be shakier than we like to assume. A lot of AI governance quietly depends on three assumptions. That the people in charge will behave reasonably. That the public will notice when they don’t. And that institutions will be able to push back. I’m not sure those assumptions deserve as much confidence as we give them.

Will we still decide?

Suppose we do notice. The next question is whether we’ll still have the authority, and the ability, to act. I think three forces matter here: measures that slowly replace the things they were meant to measure, competition that makes slowing down feel impossible, and a philosophy that could make the eventual outcome seem acceptable.

Measures that replace their purpose

The first force is about values rather than skills, and I find it unsettling. Think about a recommendation system. It predicts what you might like. You respond, and your response becomes the evidence for its next prediction. After a while, the system isn’t simply describing your tastes. It’s helping to shape them. Prediction and reality start feeding into one another.

Now imagine that happening at the level of institutions. If we measure progress mainly through productivity and automated performance, those measures influence where money goes, what gets taught and which skills we invest in. Institutions adapt around automation, which produces more evidence that automation works. Meanwhile, things that don’t show up in those measurements, such as independent judgement, relationships and taking part in decisions together, become harder to justify. Over time, delegation becomes the norm, and it’s human involvement that has to explain why it’s still necessary.

A new universal equivalent

That brings me to an idea from Marx. He called the “universal equivalent” the commodity against which all others are measured. In his account, money came to occupy that role. Things have prices, and it’s easy for a price to start standing in for a thing’s value.

I wonder whether something similar could happen with AI tokens. A token is, roughly speaking, a small unit of text that an AI model processes, and companies often charge for them. It sounds like an obscure technical detail, but it’s becoming a prominent unit for measuring AI activity. Deloitte, in a guide to AI economics, says the economics of AI “require a new currency: tokens”.

That way of thinking is already spilling into how some companies measure people. Meta reportedly ran an internal leaderboard called Claudeonomics. It tracked token usage across around 85,000 employees and ranked the top 250. People started calling the behaviour “tokenmaxxing”. Meta’s chief technology officer later reportedly told staff that token usage alone isn’t a measure of impact (InfoWorld).

That’s Goodhart’s law again, the point I made in one of my earlier essays. Using tokens is an input, not an outcome. But it’s easy to count, and it looks like activity, so the measure starts edging out the purpose. You end up encouraging engineers to consume tokens rather than understand what they’re building.

There’s another development. In March, at a forum in Beijing, an official from China’s National Data Administration announced that the official Chinese term for “token” would be ciyuan, combining the characters for “word” and “currency”. He described the token as a “settlement unit” (South China Morning Post). That doesn’t make tokens money in the ordinary sense. But it shows how easily a technical unit of computation starts being discussed in economic terms, and that’s worth watching.

Who is holding the wheel?

The second force is the race itself, and this is where I feel we’re being held to ransom. Most of the big AI labs present themselves as the careful one in a race they say they can’t afford to leave. The argument goes: if we slow down, someone less careful gets there first, so we have to keep going. There may be some truth in that. But look at what it does. Nobody feels able to stop alone, and the public is asked to accept risks it never chose. Stuart Armstrong, Nick Bostrom and Carl Shulman modelled this kind of problem in a paper called Racing to the Precipice. If being first offers enough advantage, everyone has an incentive to move faster and accept more risk, even when every competitor would prefer a safer race.

Anthropic is one of those labs, and it makes the same argument. I don’t think it should be exempt from criticism.

Listen to the language some of the people leading these companies use. When a chief executive says they’re building the most important technology in human history, and that it will lead either to paradise or catastrophe, they’re also positioning themselves as one of the few people entitled to shape everyone else’s future. I find that a grandiose self-image, and it’s fair to ask what that image might be protecting them from having to confront.

Jem Bendell, a professor who writes about societal collapse, takes the argument further. He suggests that some technology elites’ enthusiasm for acceleration can serve as psychological cover, a way of avoiding difficult questions about their own values. He sometimes uses clinical language that I wouldn’t apply to named individuals. But I think the broader pattern deserves attention. Bendell also quotes Naomi Klein making a similar point: that some billionaires don’t seem content to build arks to escape a flood. They seem determined to create the flood.

Bendell argues instead for what he calls decelerationism: slowing AI deployment so its risks can be assessed, slowing economic expansion, and making room for a slower pace of life. His concrete proposals are fairly modest, including a pause on self-authoring, autonomous AI and international limits on its use in policing and warfare. He also suggests that, if we stopped to think about it, many of us would already be decelerationists. I find that persuasive. The question isn’t just how fast we can go. It’s who gets to decide the speed.

From unnecessary to desirable

The third force is the strangest: the idea that being replaced might be desirable. This is where AI successionism comes in, and it deserves a fair hearing.

First, a distinction. Being unnecessary isn’t the same as being powerless. We could live in a society that no longer needs most human labour while still preserving democratic power, ownership and relationships. That could be an extraordinary achievement. The danger comes when economic redundancy gradually becomes political redundancy: when the systems that govern us no longer need our judgement, and we lose the practical ability to challenge them. That’s the transition I’m worried about.

Daniel Faggella describes a “worthy successor”: a posthuman intelligence so capable and morally valuable that you might prefer it to humanity as the thing that determines life’s future. To be fair to him, he rejects the label “successionism” for his own position and prefers “stewardship”. He also argues against racing towards such an outcome or building it recklessly. But he expects humanity either to fade or to transform. Richard Sutton, a Turing Award winner, said in a 2023 talk that we shouldn’t fear succession or resist it, but prepare for it. These are serious thinkers, and I don’t want to caricature their views.

The hypothesis I keep coming back to is this. I’m not saying these ideas cause disempowerment. I’m asking whether the relationship might run the other way. What if human participation becomes structurally less necessary first, and then a philosophy grows up around that reality, explaining why it’s fine, maybe even good? That would be a justification after the fact. History has plenty of ideas that didn’t create a social order but helped people sustain one. Could we gradually surrender the practical conditions of self-determination, and then learn to describe that surrender as progress?

This is a hypothesis, not a finding. A simpler explanation is that growing AI capability independently drives both automation and sympathy for succession, without either causing the other. How would we tell the explanations apart? We could check whether organisations keep independent ways to verify automated decisions, whether professions keep training the next generation, and whether attitudes to succession differ between more and less automated societies. Those are questions worth studying.

As for where I stand, I don’t think humans have to remain supreme. If artificial minds ever have genuine experiences and interests, those interests should count. My claim is narrower. People who already exist have a legitimate stake in shaping their own future, and that stake shouldn’t evaporate because we can imagine beings who might be more capable, or capable of richer experience. Someone who wants to maximise total future wellbeing might disagree, and that deserves a real argument rather than a dismissal. But that’s exactly the kind of disagreement we should still be able to have together.

Would the life be worth living?

Now the question that is, for me, the hardest of the lot. Let’s imagine we get all of this right. We keep control, the benefits are shared fairly, and we get real abundance. Would a life in which we no longer need to think, struggle, create or participate quite so much necessarily be a better life?

Achieving and becoming

Think back to the apology letter. Imagine you’ve hurt somebody. You know you need to apologise, but you’re struggling to find the words, partly because you haven’t yet worked out how you really feel about what happened. So you start writing. You delete a sentence. You try again. And somewhere around the fourth attempt, you realise something about your own behaviour that you didn’t understand when you began.

Now imagine asking an AI to write the letter. Chances are it produces something better on the surface. Warmer. More balanced. Maybe even more likely to repair the relationship. You read it, agree with every word and send it. There’s nothing inherently wrong with that. But something important has changed. The task wasn’t only to produce an apology. It was also to arrive at an understanding you didn’t have before. The struggle to find the words was part of the experience itself.

So much of our idea of technological progress is built around outcomes: faster, cheaper, more accurate, less effort. Those are real achievements, especially when the effort removed is dangerous, degrading or unnecessary. But those measures tell us little about what an activity does to the person doing it. Some things are valuable because of what they produce. Others are valuable, at least partly, because of what happens to us while we do them.

Think about a student struggling with a mathematical proof. They’re not only trying to get the answer. They’re developing the ability to reason and to sit with uncertainty. Someone learning to paint isn’t just producing images; the way they see the world changes. Caring for another person isn’t simply a list of tasks. It’s learning to recognise someone else’s needs and to stay present when there’s no easy fix. We aren’t finished people who use tools to reach fixed goals. What we do changes who we become, so what we hand over matters too.

I don’t want to romanticise difficulty. There’s nothing noble about needless suffering, and an enormous amount of human progress has come from freeing people from burdens that crushed them. Delegation can open doors for people who would otherwise be shut out. The real question is whether we can tell the difference between burdens that diminish our lives and activities through which our lives gain depth.

Why bodies matter

So why does taking part matter so much? Part of the answer may lie in the kind of beings we are. The philosopher Hans Jonas argued that a living thing isn’t merely matter carrying out clever operations. It has to sustain itself, it depends on the world around it, and things matter to it because its own existence is at stake. I’d add that human thinking doesn’t happen in some detached processor either. It develops through the ongoing interaction between a brain, a body and a world.

That doesn’t settle whether a machine could ever have experiences of its own, and it certainly doesn’t give biology a monopoly on moral worth. But it does challenge the assumption that intelligence can be understood apart from the conditions in which it’s exercised, or that greater cognitive performance must mean a better form of existence. An AI might name every tree in a forest and predict the weather. But standing there yourself, feeling the temperature drop under the canopy, adjusting to uneven ground, noticing where your attention goes: that isn’t just a bundle of information.

There’s another tension here. We’re building systems that can operate almost continuously, at speeds we can’t match. We’re biological creatures. We need sleep, rest, time to recover, time to form relationships and time to make sense of our experiences. These aren’t flaws in our design. They’re part of what we are. Yet the relentless pressure to accelerate increasingly treats those limits as obstacles to overcome, rather than conditions to respect.

I wonder whether this helps explain some of the appeal of freeing intelligence from the body: a mind that doesn’t get hungry, tired or ill, and isn’t bound by biological rhythms or mortality in quite the way we are. Even when you put an AI inside a robot, it doesn’t share the long evolutionary history that shaped living things. So when someone suggests such a mind should inherit the future, I hear an echo of an old idea: that mind should rise above body and earth. It’s a kind of dualism, and I worry it can carry a quiet disregard for creatures whose lives are inseparable from their bodies. I’m not saying machines could never matter morally, and I’m not saying everyone who wants faster AI development believes this. I’m questioning the assumption that what matters most about us is only the part that can be copied.

Maybe the problem isn’t that human beings can’t keep up. Maybe it’s that we’ve stopped asking whether the pace we’re demanding is compatible with the kind of lives we want to live.

A world that answers back

The sociologist Hartmut Rosa offers a useful word for what’s at stake: resonance. A meaningful relationship with the world isn’t just about controlling or consuming it. It’s about being affected by something, responding, and perhaps being changed in return. You can’t fully manufacture that, because it depends on the other thing remaining a little independent of you. This is part of why I’m wary of AI systems that are very good at imitating connection. A companion that remembers your preferences and is always available can help, especially when someone is lonely. But a relationship built mainly around keeping you satisfied may lack the mutual vulnerability that gives human relationships their weight.

We’re beginning to see what handing over those experiences can look like. Anthropic studied 1.5 million real conversations with its own product. Most showed no meaningful cause for concern. But among the rare patterns researchers flagged were people asking AI to write highly personal, value-laden messages in full, then apparently sending them word for word. In other words, skipping the fourth attempt at that apology letter. The researchers stressed that users weren’t simply being manipulated. They had asked for those messages and generally accepted the responses without challenging them. The study measures potential disempowerment, not confirmed harm. But it gives a concrete example of what voluntary handover might look like.

The time we were promised

Then there’s the promise that technology gives us time back. Bendell points out that this hasn’t historically worked out as advertised. Washing machines were supposed to free people, mostly women, from hours of housework. They did save time, at least at first. But expectations changed, and people began washing clothes far more often. Email replaced letters, but it also filled our days with a steady stream of new demands. Time saved by technology often gets absorbed by extra output, consumption or simply keeping up.

Bendell draws on sociologists such as Judy Wajcman, who argue that this isn’t just a quirk of particular technologies. Innovation happens inside an economy that keeps asking for more, so every efficiency becomes a new expectation. His point is that if life feels as though it’s speeding up, that may be because our systems are organised to make it speed up.

AI looks like another turn of that wheel. If an agent can write the code in an afternoon, I don’t necessarily get my afternoon back. More often, the business pulls tomorrow’s work into today. So when I ask whether the life that’s left will be worth living, I’m not only asking how much work remains. I’m asking whose time we get back, and who decides how we spend it.

The best objection

There’s a serious objection to all this. Who gets to decide what counts as meaningful participation? If someone freely chooses a life of comfort, leisure and technology, who am I to call it deficient? History is full of people in authority forcing supposedly “meaningful” work onto others who would rather have been free of it. Rest matters, and doing very little can be part of a good life. Arguments that celebrate effort can also be selective about who has done all the exhausting work.

So I’m not arguing that people should be made to struggle. My point is narrower. A good society should preserve the conditions in which different kinds of meaningful participation remain genuinely possible. The freedom to delegate matters, but so does the freedom to do something yourself, as a real option rather than a museum piece. And individual willpower isn’t enough to protect that freedom, because participation can disappear quietly. A school begins assessing finished work rather than understanding. A workplace automates routine decisions and leaves fewer chances to develop judgement. Each step seems sensible, and no individual can be expected to resist practices that everyone around them has adopted.

Being concerned about this doesn’t mean being against technology. Bendell calls himself a technophile, and he suggests “post-Luddite” as a better label for someone who wants restraint: not opposed to tools, but opposed to letting one narrow story of progress decide everything for us.

So the question isn’t how much of life’s work we can be spared. It’s whether, in sparing us from it, we also remove too many of the experiences through which life becomes our own.

Closing: the question I can’t stop asking

And that brings me back, one last time, to the apology letter.

I don’t think the answer is to refuse AI on principle. But I don’t think it’s to hand over the whole task either. There’s a third way. AI could help you work out what you’re struggling to say. It could ask how the other person might have felt. It could help you find your own way to an honest sentence. Maybe the letter would still come out better. But the real gain would be understanding something about yourself that you hadn’t understood before. The technology would be helping you become, rather than doing the becoming for you.

That’s a small example of a much bigger choice. I said at the start that we’re on a default path, so let me be precise. I’m not predicting inevitable doom, and this doesn’t need a villain. Nobody has to decide to disempower humanity. It can happen when every person, team, company and government does the locally sensible thing, while nobody takes responsibility for the whole. Changing course means deciding that some friction is worth keeping, that some expertise is worth paying for, and that some participation is worth protecting, even when the dashboard says it isn’t efficient. And if I’m right that nobody is really at the wheel, the first challenge is working out how we take it together. The authors of the gradual disempowerment paper say nobody has a concrete, plausible plan for preventing it. I don’t have one either.

That doesn’t mean AI can only make things worse. It could make each of us far more capable. What matters is whether that capability gives people more influence over the decisions that shape their lives, or whether it stays with the individual while institutions gradually stop needing them.

If we want a different outcome, there are at least four things worth protecting. We need ways for beginners to become experts, even when machines could do the work faster. We need human approval that means something, rather than becoming a rubber stamp. We need to measure understanding and resilience alongside output. And we need to ask of every tool whether it builds human capability or quietly takes over its exercise.

Slowing down can sound like retreat, but I think it may be part of the answer. Bendell describes it as both a policy demand and a personal discipline. As policy, that could mean restraint on the most autonomous AI systems, limits on their use in policing and warfare, and a pace of deployment that gives society time to assess risks before they become locked in. As a personal discipline, it means noticing when we’re rushing and asking who benefits from the rush. Neither is enough on its own. But both offer ways to reclaim some say over the direction we’re travelling.

I started by saying that AI has left me feeling hollowed out. I no longer think that’s just a software engineering problem. I think it may be an early sign of a much wider drift.

Through all of this, I’ve kept returning to three questions. Would we notice if we were losing control? Would we still be able to decide what happens next? And would the life that remained be worth living? On the first, the evidence is mixed. On the second, I can see the mechanisms and some early warning signs, but I don’t know how quickly they might unfold. On the third, I think the answer depends on choices we haven’t made yet, and on whether we recognise those choices while they’re still ours to make.

I don’t think we need to defend human supremacy. I’m not nostalgic for scarcity, or for hardship that served no purpose. What I want to preserve is a shared world where people can understand the systems around them, question them and have a real hand in shaping them. A world where effort, surprise and encounters with other people are still ours to choose.

We’ve become extraordinarily good at imagining technologies that can do more and more for us. Maybe it’s time we became just as thoughtful about what we want to keep doing, experiencing and becoming for ourselves. Progress is always progress towards something, even when nobody has stopped to ask what that something is.

So I’ll leave you with the question I can’t stop asking. If we build a world that no longer needs us to do very much, what will we have become?

Sources