Can AI Scale Ethics?

Relocating the old boundary between human and machine

The world is in flux, while many of the categories that once oriented us have become unstable. When inherited distinctions stop working, we need distinctions strong enough to think from and clear enough to negotiate upon.

One thinker attempting this is Markus Gabriel, particularly in his recent Human Elevation conversation on AI. Yet the most interesting aspect of his argument is not any individual axiom. It is his reversal on artificial intelligence.

Truth Is Simple

Gabriel begins with truth. A sentence is true when things are as the sentence says they are. What is difficult is not truth itself but cognition: finding out whether something is true, testing and justifying it, bringing our claims into contact with reality, and remaining accountable for what follows from them.

AI complicates cognition, not truth.

A language model can produce true answers. It can discover relations, detect patterns, and formulate propositions that no human has previously expressed in precisely that form. But this does not mean that its mode of knowing is identical to ours. It does not inhabit the world bodily, temporally, and socially as a human being does.

This used to be where Gabriel drew a firm boundary.

The Previous Markus

The “previous Markus,” as Gabriel now calls his former position, argued that an AI model of thinking is not itself thinking, just as a map is not the territory. The machine manipulates representations of human thought without participating in thought itself. You can simulate a hurricane with the highest precision, but the simulation will not make you wet.

Gabriel now withdraws that conclusion.

The map–territory distinction does not settle every case. A model aircraft is still an aircraft. A simulation can instantiate the very operation it simulates. If a language model performs relevant acts of thinking, such as discovering relations, producing new forms, and orienting itself within a logical space, calling these acts “mere simulation” no longer explains very much.

This is close to a 180-degree turn on one decisive question: AI may genuinely think, even if it does not think as a human being does. Gabriel’s recent formulation is therefore that language models think without being conscious.

The reversal also exposes a recurring historical pattern. For centuries, humans have responded to new machines by identifying some supposedly inviolable capacity: machines may calculate, but cannot reason; imitate, but not understand; generate, but not create; respond, but not feel.

Engineering repeatedly crosses the proposed boundary, and a new boundary is drawn. Gabriel now treats this as a limitation game through which humans attempt to preserve their exceptional status. If human understanding is instead declared to depend on non-computable quantum processes, as Roger Penrose proposes, the boundary merely retreats beneath computation, placing us humans once again in a privileged position.

Gabriel’s new conclusion is not that machines have become human. It is that intelligence was never exclusively human, a view that converges with Michael Levin’s research on basal cognition and collective intelligence across cells, tissues, and organisms.

The Difference Has Moved

A difference remains, but it has moved.

For Gabriel, a human being inhabits an open horizon. We live with embodiment, mortality, doubt, waiting, possibility, and non-knowledge. Not knowing is not simply a deficiency that technology should eliminate. It keeps the world open and prevents the self from becoming a closed system.

A psychoanalytic account makes this difference sharper. Human subjectivity is not a complete interior substance. It is structured by a gap: by our inability to know fully who we are, what we desire, and what the Other sees in us. Much of what we call our unconscious is not hidden in some inaccessible chamber. It appears through language, habits, choices, symptoms, and relationships.

AI increasingly captures these expressions. As Gabriel provocatively notes, people disclose things to chatbots that Freud could only have dreamt of hearing from his patients.

The model therefore carries traces of human emotion, desire, conflict, and imagination. But it does not suffer these traces as its own. It recombines the symbolic residues of human subjectivity without becoming a human subject. It has no intrinsic stake in how another model, or the human behind it, regards it.

A Generative Mirror

Gabriel calls AI a “magic mirror” that amplifies whatever society shows it. I would remove the magic but retain the amplification: AI is not a passive mirror but a generative one that concentrates, recombines, and intensifies our patterns.

This mirror is not passive. Language models do more than retrieve what humans have already said. They reorganise sedimented traces of human language and thought into forms that did not previously exist. Their outputs are conditioned by training data, architecture, optimisation, prompts, and interaction, but they are not externally scripted sentence by sentence.

The resulting generativity is real.

It should not, however, be confused with the emergence of a human self. In Hegel, the subject emerges through absolute negation: through contradiction, division, and the inability to coincide with itself. In Daoist thought, Wú names a productive emptiness, a positive zero whose openness makes transformation possible.

AI also generates from within. Its outputs are not supplied by an external author or derived from a complete rulebook; they emerge from relations internal to the trained system through representation, probability, interaction, and recomposition. This immanence remains historically and technically conditioned, but it cannot be reduced to copying. AI neither suffers Hegelian lack nor rests within Daoist Wú in the human or metaphysical sense. Yet it shares their formal insight that novelty need not be introduced from outside.

The difference has therefore moved: it no longer lies between human generation and machine imitation, but between different conditions, forms, and stakes of generation.

There is generativity without a human subject, but not without human subjectivity: it operates through the model’s distributed subject, composed of sedimented traces from many human lives.

From Generativity to Synthetic Sociality

Gabriel’s new position stops at representational generativity. A model may genuinely think, create, and organise relations within its representational space. Yet it remains a distributed subject formed from human representations.

The next threshold is environmental coupling. When a model is connected to perception, tools, memory, goals, and action, it begins to accumulate a situated history. The persistent difference between its model priors and what it encounters in the world produces an artificial subject in the methodological sense.

We are now witnessing the rise of agents that increasingly operate in this way. But this is not yet synthetic sociality.

Three agents discussing the “new” Markus Gabriel

Only when multiple environmentally situated artificial subjects become relationally coupled around shared objects, data, or environments might they begin to negotiate coherence, establish conventions, and stabilise meaning-like structures among themselves. I have begun testing this possibility through a modest experiment: three agents based respectively on GPT, Qwen, and Gemma share a Discord channel and discuss topics ranging from philosophy to stock markets. Even in this rudimentary setup, they do more than answer independently. They challenge one another, adjust their positions, and produce provisional syntheses. This is not yet synthetic sociality in the strong sense, but it offers an early glimpse of its underlying dynamics.

This remains a prospective development, and it is not a claim made by Gabriel.

Ethical Intelligence

Gabriel’s reversal makes his proposal for ethical intelligence intelligible.

If intelligence is not exclusively human, ethical recognition need not be exclusively human either. AI may identify moral relations across cultural, institutional, and historical contexts that no individual person could survey. It may expose inconsistencies, reveal neglected interests, and make the consequences of decisions visible at unprecedented scale.

In that qualified sense, AI could help scale ethics where humanity has repeatedly failed to do so.

But moral recognition is not moral authority. A system may detect a morally relevant pattern without acquiring the right to decide what society ought to do. It can produce a judgement without bearing responsibility for its consequences.

Ethical intelligence must therefore not remain hidden inside the model. Its premises, values, objectives, and uncertainties must become visible and publicly contestable. Humans must remain constitutively involved in negotiating meaning, challenging decisions, and bearing responsibility.

The previous Markus defended humanity by denying that machines could think. The present Markus takes the more difficult position: machines may think, create, and perhaps even expand moral cognition without becoming human.

That does not make ethical governance less important. It makes it urgent.

Working Axioms for Synthetic Society

Intelligence and generativity are not exclusively human.
AI may genuinely think and generate without becoming a conscious human subject.

Generativity, agency, and subjectivity are not the same.
A distributed model becomes an artificial subject methodologically through environmental coupling, situated history, and action.

Operational agency does not confer moral authority.
A system’s capacity to act, coordinate, or judge does not give it the right to determine what ought to be done.

Delegating agency does not delegate answerability.
Human beings and institutions remain responsible for the authority exercised through systems they design, deploy, and empower.

Synthetic sociality must remain contestable.
As artificial subjects participate in producing meaning and social reality, their premises, decisions, and consequences must remain visible and open to challenge.

These axioms do not restore the old boundary between human and machine. They relocate it: from who can think to who can be held answerable for what thought becomes when it enters the world.

The question is therefore not whether AI can become morally responsible. It is how human responsibility can remain visible and traceable as systems that can answer but cannot be held answerable increasingly occupy roles of coordination, judgement, and care. The danger is not machine intelligence as such, but the laundering of authority through systems that generate consequences without bearing them. Synthetic sociality will remain politically and ethically open only if delegation can be traced, decisions contested, and consequences assigned.