MH
← All essays
Originally on Substack

When Progress Makes Humanity Optional

AI successionism, gradual disempowerment, and how we might come to justify our own obsolescence.

We tend to imagine humanity’s displacement by artificial intelligence as something that happens against our will. Perhaps a superintelligent system escapes our control, develops objectives incompatible with human survival, or simply comes to regard us as an obstacle. These scenarios dominate much of the discussion about existential risk, but I’ve become increasingly interested in a quieter possibility.

What if we gradually construct a civilisation that no longer depends on meaningful human participation? One in which we remain comfortable and prosperous, but increasingly lack the understanding or authority to influence the systems responsible for our existence. Rather than resisting this transformation, we might eventually come to regard it as progress.

This is what interests me about AI successionism, the idea that artificial intelligence could, and perhaps should, eventually succeed humanity. My concern isn’t primarily with the relatively small number of people who explicitly advocate some version of this future. It’s that the economic, technological and cultural forces already shaping AI development might gradually make such a future seem reasonable, even to people who would reject it today.

We might never consciously decide that humanity should become obsolete. We might simply build a world in which human participation becomes less necessary, then adjust our understanding of progress to accommodate what we’ve built.

The philosophy of succession

One of the more interesting contributions to this discussion comes from Daniel Faggella, who has written extensively about creating what he calls a worthy successor. He imagines a future intelligence possessing qualities that might justify entrusting it with the long-term future of life.

Faggella explicitly rejects the label successionism for his own philosophy. In an update to his essay on 15 October 2026, he describes his position in terms of stewarding and expanding the living process, including through biological and neurological transformation. He also argues against a reckless AI takeover and supports slowing AI development. His position deserves to be distinguished from simply wanting machines to replace people.

Nevertheless, it raises some of the same fundamental questions. Faggella argues that humanity cannot expect to remain in control indefinitely. Biological and technological change will continue, and he sees our responsibility as helping to ensure that whatever succeeds us is worthy of carrying forward what matters about life.

His argument isn’t simply that greater intelligence means greater moral worth. He places considerable importance on consciousness, self-creation and the capacity for life to flourish. These are serious philosophical considerations. If artificial beings eventually possess genuine consciousness, relationships, creativity and morally significant experiences, I see no obvious reason why their interests should be dismissed merely because they aren’t human.

Where I struggle with the argument is the assumption that humanity’s succession is inevitable, or that creating something more capable necessarily changes what humanity ought to do with its own future. Our species will undoubtedly change, and nothing guarantees that humans will remain the most capable intelligence indefinitely. But change isn’t the same as surrendering meaningful authority over our collective existence.

There’s also a distinction between recognising the moral value of future beings and deciding that existing beings should give way to them. Even if artificial minds could eventually experience vastly greater wellbeing than humans, that wouldn’t automatically settle the claims of people who already exist.

Someone committed to maximising total future wellbeing might disagree. From that perspective, a future containing vastly more valuable conscious experience could be preferable, regardless of whether humans remain central to it. I think that argument deserves a proper response rather than an appeal to human exceptionalism.

My own position is that existing people have legitimate claims to participate in shaping their future. Those claims shouldn’t simply disappear because we can imagine beings that might accomplish or experience more.

Coexistence, transformation and succession are different possibilities. The technological feasibility of one doesn’t establish its ethical necessity.

Disempowerment without a successor

The more immediate concern, however, isn’t the philosophy of succession. It’s the gradual erosion of human influence over civilisation.

In 2025, Jan Kulveit and colleagues published Position: Humanity Faces Existential Risk from Gradual Disempowerment, examining how incremental advances in AI could undermine human influence across the economy, culture and government. Their argument is particularly interesting because it doesn’t require an AI system to become hostile or deliberately seize power.

Much of our existing influence comes from the fact that institutions depend upon us. Businesses need human workers and customers. Governments depend on citizens, taxation and political legitimacy. Cultural institutions depend on human creators and audiences. These relationships are far from perfect, and dependence has never guaranteed justice or meaningful autonomy. Nevertheless, it gives institutions practical reasons to remain responsive to human interests.

As artificial systems become capable of performing more economic, administrative and intellectual functions, some of those dependencies could weaken. An organisation that no longer needs much human labour may have fewer economic reasons to accommodate workers. A government whose administrative capabilities increasingly depend on automated systems may find itself relying on expertise it cannot independently reproduce or challenge.

The important point is that none of this requires anyone to intend the eventual outcome. Each decision can be justified on its own terms. A business reduces costs, an institution improves efficiency, or a government automates a difficult administrative process. Yet these decisions can interact in ways that gradually redistribute power away from the people affected by them.

This concern has a longer intellectual history. Nick Bostrom explored how competitive pressures might eliminate forms of existence we value without anyone explicitly choosing that result. Paul Christiano described a possible future in which optimisation steadily weakens human influence without a dramatic takeover.

I don’t want to present these ideas as though they’re new. What interests me is how such structural changes might eventually influence the values through which we interpret them. If human participation becomes less important to the functioning of our institutions, might we also become less inclined to regard that participation as essential to progress?

When automation becomes dependence

Consider a software engineering organisation. Initially, AI generates code while human developers review and approve the changes. As generation becomes cheaper and faster, human review becomes the bottleneck, so the organisation introduces automated review. Eventually, approval itself becomes an obstacle to throughput, and routine changes are approved automatically.

There’s nothing inherently wrong with this progression. Automated testing and formal verification can provide stronger assurances than manual inspection, and plenty of human review is superficial or ineffective. The problem arises when we consider how the organisation maintains the expertise required to supervise these systems.

Experienced engineers develop judgement through ordinary work. They debug failures, misunderstand requirements, investigate unexpected behaviour and learn from mistakes. They gradually acquire the ability to recognise when something is wrong, including when the available evidence initially suggests everything is fine.

If more of this work becomes automated, organisations may find themselves relying on expertise accumulated by previous generations while weakening the processes through which new engineers develop comparable skills.

This problem predates modern AI. In her 1983 paper Ironies of Automation, psychologist Lisanne Bainbridge described how automating routine activities can leave human operators responsible for exceptional situations while depriving them of the regular practice needed to handle those situations competently.

What concerns me is the possibility of a self-reinforcing dependency. As automation reduces the immediate need for human expertise, organisations invest less in developing it. Independent scrutiny becomes harder and more expensive, which makes further automation attractive. Eventually, understanding the system may itself require consulting the system.

An organisation can preserve formal human responsibility while gradually losing the practical capabilities that make responsibility meaningful. Someone still signs off the decision, but their understanding depends almost entirely on explanations generated by the infrastructure they’re supposed to supervise.

This outcome isn’t unavoidable. AI can also help people develop expertise, make unfamiliar systems easier to understand and strengthen independent verification. Organisations can preserve apprenticeship, testing, auditing and meaningful human intervention.

The difficulty is that these things impose costs, while the immediate benefits of automation are often easier to measure. Maintaining a workforce capable of independently challenging automated systems may appear inefficient right up until that capability becomes essential.

This raises a question that extends beyond software engineering: what happens when the values we use to optimise institutions begin to shape what those institutions consider valuable?

When the map begins to make the territory

Jean Baudrillard’s concept of the precession of simulacra has influenced how I think about this problem. Baudrillard described a world in which representations no longer merely reflect reality but increasingly precede and organise it, eventually producing a hyperreality whose relationship with an independent referent becomes unstable.

I don’t think modern AI reproduces his argument exactly, but it introduces a related phenomenon.

Consider a recommendation system that attempts to understand someone’s preferences. It observes their behaviour, predicts what they might enjoy and recommends information, entertainment or activities accordingly. The person responds to those suggestions, and their responses become evidence for subsequent predictions.

Over time, the system is no longer simply describing preferences that exist independently of it. It’s participating in the environment through which those preferences develop.

Unlike Baudrillard’s hyperreal, the referent hasn’t disappeared. The person’s preferences remain real, but they are partly shaped by the system measuring them.

Other thinkers have described this mechanism more precisely. Philosopher Ian Hacking explored looping effects, in which classifications influence the people being classified, whose subsequent behaviour changes the classifications themselves. Machine-learning researchers, including Juan Perdomo and colleagues, have examined performative prediction, where predictions influence the distribution of outcomes they attempt to predict. Donald MacKenzie’s An Engine, Not a Camera makes a related argument about financial models becoming active components of the markets they supposedly represent.

What these ideas have in common is that a model can participate in creating the reality it subsequently describes. It might even become increasingly accurate, while the world it measures becomes increasingly shaped by its earlier predictions.

“Language is a virus from outerspace” — William Burroughs

This becomes more consequential when we extend the mechanism from individual preferences to institutions and culture.

Imagine organisations increasingly measuring progress through productivity, computational capability and the performance of automated systems. These measures influence investment decisions, how work is organised, which skills are taught and which technologies receive support. The resulting institutions become better adapted to automation, producing further evidence of its economic advantages.

Meanwhile, activities that don’t fit those measures as neatly, such as independent judgement, human relationships, curiosity or participation in collective decisions, can struggle to demonstrate their value on the same terms.

The feedback loop is therefore potentially cultural as well as technical. As people become accustomed to institutions in which machine decisions are normal and automated output is authoritative, their expectations can change. Delegation no longer seems exceptional. Instead, it is human intervention that increasingly requires justification.

This is where I see a possible connection between institutional disempowerment and the gradual acceptance of posthuman values.

The systems we build don’t merely perform tasks according to our existing priorities. They can change the environments in which those priorities are formed. Over time, we may start to regard the values embedded in those systems as natural or self-evident, precisely because so much of the world has been reorganised around them.

That doesn’t mean people will inevitably embrace the idea of humanity’s replacement. It does suggest a mechanism through which changing institutions might influence what people come to consider desirable.

We become what we behold. We shape our tools, and thereafter our tools shape us. – DX Innovation Institute

From dispensability to desirability

There’s an important distinction between making human participation economically unnecessary and making human existence morally undesirable. I think this is where arguments about AI succession often become confused.

An automated economy could create extraordinary opportunities for human flourishing. People might spend less time on work they dislike and more time on relationships, education, creativity or public life. Being liberated from economically necessary labour isn’t the same as losing the ability to shape one’s future.

The concern is what happens if automation also weakens the institutions and capabilities through which people exercise meaningful influence.

As human participation becomes less central to economic and administrative systems, automated decision-making may become the default. Organisations adapt to this arrangement, and cultural narratives evolve alongside it. What initially required justification becomes ordinary, while the continued involvement of humans starts to look like an unnecessary constraint.

At some point, arguments that humanity should eventually relinquish its central role might encounter a society already accustomed to treating human participation as optional.

This is the possibility I find particularly interesting: successionist ideology could emerge as a retrospective justification for structural changes that have already diminished humanity’s role in determining its future.

It wouldn’t necessarily be the ideology driving the transformation. It might instead provide a philosophical explanation for why that transformation should be considered desirable.

Of course, successionist ideas already exist among philosophers, technologists and AI researchers. I’m not suggesting they originate only after society becomes disempowered. The question is whether those ideas might gain wider cultural acceptance as our relationships with automated systems change.

There’s a familiar pattern in human history of institutions developing explanations that legitimise arrangements already in place. An ideology needn’t create a social order to help sustain it.

But there’s also an important alternative explanation. Increasing AI capability could independently encourage automation and sympathy for AI succession, without either causing the other. The relationship would need to be demonstrated rather than inferred from the order in which changes occur.

For now, I think of this as a hypothesis about how technological dependence might influence cultural values, rather than a prediction about where civilisation must end up.

The comfortable loss of agency

The future I find most troubling isn’t necessarily one in which humanity disappears. It’s one in which people remain alive, comfortable and perhaps even happy, but lose meaningful influence over the conditions of their existence.

Imagine a civilisation where artificial intelligence provides excellent healthcare, personalised education, abundant entertainment and most productive labour. Daily life becomes easier, material insecurity declines and people experience fewer frustrations. These would be genuine achievements.

But suppose people also become increasingly dependent on systems they cannot meaningfully challenge, redirect or replace. Their preferences are anticipated and satisfied, while their influence over the institutions shaping those preferences steadily diminishes.

They might reasonably describe themselves as freer than previous generations. Yet the freedom to choose among available options isn’t necessarily the same as the freedom to influence which options exist in the first place.

I’ve previously thought of this as a form of domestication, although the analogy has obvious limitations. Dependence isn’t inherently domination. Modern civilisation already relies on specialised knowledge and responsibilities delegated to others. Nobody needs to understand the entire electricity grid or practise their own medicine to exercise meaningful autonomy.

The more useful question is whether delegation remains revocable, contestable and subject to human authority.

Can people challenge consequential decisions and obtain genuinely independent review? Can communities influence the objectives of systems that shape their lives? Can institutions investigate failures without depending entirely on the systems being investigated? And can people pursue lives that aren’t organised primarily around what automated systems predict will satisfy them?

These questions seem more important to me than preserving human labour or insisting that people must personally perform tasks machines can do better.

My concern ultimately rests on a particular ethical commitment. I believe existing people have legitimate claims to participate in shaping their own future, and that their interests shouldn’t be treated as interchangeable quantities that can simply be offset by the hypothetical greater welfare of future beings.

This is broadly a person-affecting and autonomy-centred position, with affinities to the civic republican understanding of freedom as non-domination. It isn’t a universally accepted moral theory, and someone committed to maximising total future wellbeing may reach a different conclusion.

But that disagreement is precisely the kind of question humanity ought to retain the capacity to deliberate about.

A civilisation that has lost the ability to meaningfully participate in deciding its own future would have surrendered something important, regardless of how efficiently it operates or how generously it provides for its inhabitants.

What would count as evidence?

There are really two claims here, and I think they need to be investigated separately.

The first concerns gradual disempowerment. We can examine whether institutions retain independent verification, whether professional expertise continues to develop, and whether human decision-makers possess practical authority rather than simply formal responsibility.

An organisation in which nobody can independently investigate an automated decision would be a warning sign. So would a profession that depends on experienced practitioners but has stopped developing their successors. Conversely, evidence that AI strengthens expertise, independent oversight and public accountability would weaken the concern.

The second claim is more speculative: that declining human influence could make people increasingly receptive to arguments for AI succession.

We might investigate whether people exposed to extensive automated decision-making become more willing to transfer consequential authority permanently to artificial systems. We could examine whether institutions that depend less on human participation increasingly present that independence as a desirable end in itself, or whether attitudes towards human succession differ across societies with different levels of automation and accountability.

Any such research would need to distinguish causation from correlation. Technological capability, economic insecurity, cultural change and existing philosophical commitments might independently influence both institutional arrangements and attitudes towards succession.

The important question is whether the experience of disempowerment itself changes what people believe humanity’s role in the future ought to be.

I don’t think we currently have sufficient evidence to settle that question. But the possibility is consequential enough to deserve attention.

A future worth participating in

The issue, ultimately, isn’t whether artificial intelligence can surpass human capabilities. In many domains, it already has. Nor is it whether future artificial beings might possess genuine moral value. If they do, their interests will deserve consideration alongside ours.

What concerns me is the assumption that progress can be understood primarily through the expansion of intelligence, capability and productive output, independently of the people whose lives are affected by that expansion.

A world can become more productive without becoming more participatory. It can become more intelligent without becoming more comprehensible to its inhabitants. It can provide greater comfort while leaving people less able to determine the conditions of their lives.

I don’t believe we need to preserve human supremacy or insist that the future must resemble the present. Technological progress can be enormously liberating, and there’s little value in protecting hardship or unnecessary labour merely because they are familiar.

But I do think we need to preserve the possibility of a shared world in which humans can understand, question and meaningfully influence the systems governing their existence. That may require institutions to protect forms of expertise, independence and participation that don’t always maximise short-term efficiency.

AI successionism asks whether artificial intelligence should eventually inherit the future. Gradual disempowerment raises a more immediate question: will humanity retain the ability to participate meaningfully in deciding that?

What interests me most is the relationship between the two. Could we gradually surrender the practical conditions of human self-determination, then come to regard that surrender as evidence of progress?

The danger isn’t simply that machines may one day become capable of building a civilisation without us. It’s that we might lose the ability to shape that civilisation ourselves, and eventually come to believe that our absence from its direction is exactly what progress was supposed to achieve.

Sources and further reading