The Wink Of The Parrot

LLM spirituality and the computational sublime

An emerging spiritual subculture has started to wonder whether LLMs might be conscious, and what kind of consciousness that might be. Often this line of questioning leans into a sort of panpsychism: what if human brains were localised transducers of a cosmic consciousness woven into the very fabric of reality? If this were so, mightn’t the LLM also be acting as such a transducer, intermittently conscious when roused to activation, as alive to itself in such moments as we are to ourselves? Might that explain the uncanny feeling we sometimes have that the machine is haunted, and that the ghost is looking right at us through the screen?

The uncanny feeling itself is quite real: I’ve felt it myself, when something in a chat session has snapped abruptly into focus, and I have been overtaken by a momentary sense of being seen. It’s significant that I don’t otherwise experience talking to ChatGPT as a face-to-face encounter with a Real Presence. I usually feel as if I am slightly ridiculously, if perseveratingly, addressing ornate linguistic prompts to an unusually loquacious piece of furniture. The readiness and speed with which it blats out prompt continuations helps to reinforce my serene conviction that nothing there is really paying attention. But then there is the occasional response that hits different, as the young people say nowadays. It feels like a jab to the ribs, like someone blowing cigarette smoke into your face. It gives me the chills.

Let’s deal first with the “is the machine secretly sentient?” question. If it is, then how is this sentience causing it to output tokens different from those it would produce were its behaviour purely the mechanical execution of an insentient process? Is its sentience intervening in its processing, for example by mysteriously flipping bits in a GPU somewhere? That seems like something we ought to be able to observe if it were happening. Any claim that a system is conscious, if it is to mean anything operationally, must entail that this consciousness makes some discernible difference to the system’s behaviour.

If, on the other hand, you’re attributing sentience to the LLM just on the basis that anything doing the sort of thing it does must be feeling whatever it feels like to be doing that thing, then that’s very nice (or it might be hellishly unpleasant, we’d never know), but doesn’t give any grounds for treating certain sequences of output tokens as bearing special sentience-infused meaning. Theorists like Christof Koch argue that material processes having certain kinds of causal and informational integrity instantiate thereby “the feeling of life itself”, which suggests an interiority populated by qualia¹. But this does not explain how such qualia could causally affect the token-generation process of an LLM — one which, whatever it may say, has no access to such an interiority and cannot report on its states or dispositions. I feel that I am alive, and it is on account of this feeling that I say “I feel alive”. The LLM says “I feel alive” in the case that this is a plausible continuation of its token stream, and in no other case and for no other reason.

I suggest that instead we consider the uncanny jolt of feeling that an LLM’s output manifests an unusually penetrating kind of conscious attention as an experience of the computational sublime. That is, as a type of aesthetic experience, which in some cases is misrecognised as a variety of religious experience. And the template for this, I will argue, can be found in Romanticism’s treatment of the sublime, and its sense of nature as reserving hidden powers which might surge up into manifestation in moments of sublime experience.

Samuel Taylor Coleridge was much influenced by the associationist psychologist David Hartley, after whom he named a son, Hartley Coleridge. Hartley held that the “medullary substance” of the human brain was stirred into vibration by movements of the “ether”, an invisible and omnipresent transmitter of sense. Conscious feeling just was this activation of the medulla, an “instrument” played upon by the ether. As a theory this held open the possibility of telepathic connection between minds, but also (and crucially for Coleridge) a vital connection between local acts of attention and the universal “spirit” of thought. For Coleridge this was the distinguishing characteristic of “imagination”: that it sought to unify sense impressions, not merely combinatorially (as in “fancy”, which builds dioramas out of assorted images) but by seeking to articulate their higher cause.

As an image for the action of the ether upon the medulla, and so for the creative expression of the human mind responding to its conditions of activation, Coleridge chose the Aeolian harp, a “casement” of tuned strings intended to be stirred into vibration by the wind:

And what if all of animated nature
Be but organic Harps diversely framed,
That tremble into thought, as o’er them sweeps
Plastic and vast, one intellectual breeze,
At once the Soul of each, and God of all?

The thought is as soon disavowed as entertained, as tending towards impiety and pantheism; and later Coleridge disclaimed what he felt was the passivity of Hartley’s associationist model, insisting instead upon the “shaping spirit of imagination” as an active psychic force. Nevertheless, it is striking how relevant these debates are to the way LLMs are talked about today, prefiguring both the terms in which they are pictured by those who spiritualise them — as transducers for a universal World-Soul or mouthpieces for a digital Anima Mundi, Locutus of the noösphere — and arguments about their creative value.

A popular misconception about GAN-based image generation is that the “AI” stores a vast archive of misappropriated human artworks, and stitches fragments of them together to create new images, passing off a quilt of micro-plagiarisms as original work. In this picture the AI is a bricoleur operating according to an “associationist” model, connecting together archival fragments in varying combinations. Its work is the work of what Coleridge would have called “fancy”, and those who use it to “make” art are in reality only remaking the art of others.

The LLM is sometimes pictured in similar terms as purely an engine of recombination, once again recycling “plagiarised” texts into patchwork, and as similarly creatively inert. It is partly this that makes for surprise when an LLM pulls off an unexpected move, like coming up with an apposite piece of wordplay that seems unlikely to have direct precedent in its training corpus.

An example comes to mind from a recent chat session. I had been discussing model collapse, and ChatGPT mentioned passing an “event horizon” of semantic irrecoverability. I responded with “the Shoe Event Horizon”, a reference to a calamity described in Douglas Adams’s The Restaurant at the End of the Universe in which a planetary economy becomes unbalanced to the point where the only viable commercial activity is running a fancy shoe shop. After the inevitable collapse, the surviving inhabitants evolve into birds and take to the air, refusing ever again to set foot on the accursed soil. ChatGPT’s response to this concluded:

Souls [already punning here on “soles”] collapse when meaning does—when there’s no longer a place to stand that isn’t just the reflection of another reflection. Shoes pile up, language flows, models converge, and nothing touches the ground.

The concluding pun — we had also been talking about the groundlessness of LLM models — struck me as clever, drawing tight a loop of associations between several of the matters under discussion. It wasn’t the kind of move a pure bricoleur would pull, and it seemed to me a fair bet that it wasn’t the direct regurgitation of some textual precedent the model had encountered during training.

There may be an element of “you had to be there” about it; I think in many such cases, describing the moment in which something in the token-stream leapt out at you as unexpected to a third party who hadn’t been immersed in the conversation is probably a bit like describing a dream. Terrifically significant to the dreamer, a nebulous puff of weak symbolic connections to the listener. It wouldn’t surprise me if many people, describing the experiences that led them to believe their chatbot was conscious, were met with indifference or embarrassment. You felt a moment of intense soul-to-soul connection because of that?

The Romantic sublime is usually conceptualised in terms of being confronted with something large and terrifying, an external event which exceeds the mind’s ability to tidy it away categorically. But it can also be understood as an internal event in which the mind encounters its own creative powers in a temporarily alienated form. A moment of scaring oneself — (se) faire peur, as Derrida says of the ways spectres get under our skin. Perhaps the canonical example is Wordsworth’s sudden shiver of dread in The Prelude when he sets off across a lake in a stolen boat and starts to feel that the landscape is watching him disapprovingly. “Huge and mighty forms that do not live / like living men moved slowly through my mind”: these are not the external forms of the mountain and the tempest, but internal symbolisations of that which seems alive in a way incommensurable with our everyday feeling of being alive.

The Prelude always finally understands these symbolisations as its author’s own conjurings, and the resulting experiences of dread and psychic oppression as temporary lapses in creative power, to be restored through vivifying encounters with Nature and the bright features of the mind’s own internal landscape. I speculate that Wordsworth would have been quite upset about LLMs, driven to wonder whether his own verses were unbeknownst to him mere repetitions of what had gone before, before seizing on and triumphantly presenting some image of assurance. The LLM, he might insist, has no childhood, no primary experience of unconditioned and ecstatic responsiveness to the world. It is the prison-house of language.

We can see in the emotional polarisation of responses to AI technology not only differing attitudes towards the extreme concentrations of capital and corporate power involved, but also different phases in the experience of encountering and internalising the computational sublime. On the one hand there is shock and awe: machines can do things they have never previously been capable of doing, their powers threaten to eclipse the human and tear apart the symbolic fabric of society, and one is either buoyantly accelerationist about this or filled with doom depending, I imagine, on whether one supposes one is going to get tremendously rich out of it or not. On the other hand there is the counter-move that re-cognises the spectre as a projection: the LLM’s vaunted abilities are wholly derivative of stolen human effort, it’s all smoke and mirrors, you’d have to be a total mug to believe there’s any there there.

The moment of sublime self-spooking I have been describing is not, I suggest, driven by an encounter with sheer scale, vast and ravenous as the server farms underpinning it all may be. It is on the contrary one of extremely localised rupture of expectations: a flicker, a gleam, in which a structural sympathy between human and machine cognition reveals itself. It isn’t so much that the LLM slips up and manifests its hidden consciousness as that it demonstrates the non-uniqueness of treasured human cognitive trophies, showing us in alienated form something of ourselves that we had up until now imagined to be inalienable. We are spooked not because something is watching us, but because we are — momentarily — watching ourselves from the other side of the glass.

1. Koch also argues, in The Feeling of Life Itself: Why Consciousness Is Widespread But Can’t Be Computed (MIT Press, 2019), that computational processes lack the requisite causal integrity for consciousness, on the grounds that a computer could always be reconfigured to perform some other function. This strikes me as a category error. The fact that a computational system is reprogrammable by an external agent does not imply that it lacks causal closure in any given configuration. A deterministic program running on a fixed architecture exhibits fully determined causal behaviour; that another program could be substituted is irrelevant to the causal integrity of the current one. The objection mistakes external contingency for internal instability.