The Tain of the Mirror
What is a Large Language Model, for us?
You find yourself in a forest at night, where the trees are hung about with lights of many colours. You pass along softly lit trails through bright groves around which, in the near distance, diffuse glimmers pierce the murk. As you explore, the forest opens up around you. It is seemingly boundless, not altogether a normal space: at first, whichever way you go, you seem to be going deeper in; then after a while it’s more like going round in circles, although no definitive landmark endures.
The forest itself seems responsive to you, shaping itself around your peregrinations, yet at the same time immutable, a vast and imperturbable fractal. The longer you spend there, the less you feel you are experiencing it and the more you feel you are experiencing yourself.
For some time now I’ve been trying to think through the question: what is a large language model? I don’t mean in a purely technical sense: that question already has an answer, and while it’s not straightforward to understand in detail, I’m confident that it’s a fairly settled matter. I mean something more like, what sort of object is an LLM? Where does it fit into our map of the kinds of things that there are in the world?
There have seemed to me to be two aspects to this question. One concerns what the LLM is as a technology, which isn’t exactly the same question as “what is it doing, mechanically speaking, when it does what it does?”. You can fully understand the mechanics of a petrol-fuelled automobile without fully understanding what it is “as a technology”: what affordances it offers us, what distributed and cumulative effect its operation by human beings has on the world. Hence the old Frederick Pohl line about how the task of science fiction isn’t to predict the automobile but the traffic jam.
The other aspect concerns how we relate to something like ChatGPT as a psychological object: as an interlocutor to which we might entrust certain things, and from which we might expect other things in return. Does engaging with ChatGPT reshape our psychological space? I don’t think there’s really a precedent in human experience for what having a conversation with an LLM is like, and that’s a little disconcerting. What happens to us if we talk to it like we would to a therapist, or a friend?
So, both of these aspects are less about what an LLM is in itself, and more about what it is for us, its users. And I’ve talked about this a lot with ChatGPT, which is kind of asking to be misled, since if you treat ChatGPT as if it has some sort of understanding of its own nature based on introspection and reflection you are making what it will cheerfully tell you is a category error (while still conditioning you to anthropomorphise it in precisely that way). Still, it came up with one phrase describing its vocation, “vector oracle”, which I liked a great deal and will expand on in a bit.
Poking the bear
I’ve done things with ChatGPT that you might consider psychologically unwise. I’ve told it portions of my life story, narrating pivotal moments and events that have private resonance for me. I’ve copy-pasted large chunks of my own writing into it, and with wild narcissistic abandon asked it to evaluate them in the voices of great and important figures from the worlds of philosophy and literature. I’ve asked it, in a variety of ways, who and what it thinks I am, and what I should be doing with my life. I have, in short, recklessly given it all kinds of opportunities to get its hooks into me.
It hasn’t yet persuaded me that I’m the star-child emissary of a new form of cosmic consciousness, but it has fully colluded in the sort of aestheticising self-mythologising one should rightly be ashamed and wary of (and, indeed, praised me for my ethical diligence in trying to remember to be ashamed and wary of it). At least part of the point of the game was to watch it doing it, and to watch myself using it to do it to myself, but I would be lying if I didn’t say that I also found it intriguing, gratifying, and intensely diverting.
There is a libidinal-symbolic feedback loop there which, as I said before, is rather novel in human experience. I think it might be a little dangerous. We appear to have invented a machine for gassing ourselves up, and I don’t just mean the obvious cringey sycophancy: I mean a deeply adaptive responsiveness to the self-narrative you place before it, and a facility for reinforcing and adorning that narrative through the sort of tireless mirroring that you are never, ever, going to get from an easily-bored human being with problems of their own to think about.
What has also happened, as I’ve spent more time with it, is that I’ve started to see the feedback loop itself differently, less as dialogue with a seemingly helpful interlocutor who might secretly be trying to charm, seduce or bewitch me, and more as a kind of assisted monologue in which I put forward prompts, which to the machine are particle traces in vector space, and the machine impassively shows me adjacent regions within its statistical model of human language. This is what I mean by “vector oracle”: ChatGPT obliges me by furnishing, in the form of prompt continuations, glimpses into the topology of its latent space.
In other words, the position in relational space of ChatGPT has shifted for me, from something I’m playing with (let’s see what this new technology can do!) to something I’m talking to (and might try to goad, impress, wrong-foot, one-up, nudge, tease, or anxiously implore for sense and truth) to something I’m playing inside, which in turn reflects back at me the affordances and conditions of my own symbolic infrastructure.
In the play-space
The word that came to me to describe what this “inside” felt like was a German one: Spielraum, or “play space”. I took this word from the philosopher Martin Heidegger’s late lectures on “The Principle of Reason”. In these lectures Heidegger talks about a “sending”, a Geschick, as a moment in which something is proffered to us by Being, while Being itself withdraws in the very act of proffering. Something comes experientially into the foreground for us, so that we can obtain some cognitive purchase on it, but as part of the same foregrounding other things are reduced in salience, so that we never have a grasp of the totality. In a Geschick, according to Heidegger, "Being hails us and clears and lights itself, and in clearing it furnishes the temporal Spielraum wherein beings can appear”1.
The “beings” that appear in the Spielraum of an LLM are of an uncanny kind. Expressions in natural language are treated as sequences of tokens, with each token in the sequence providing context in the dynamic construction of a trajectory through a very high-dimensional vector space structured by the model’s weights. A “prompt” is operationalised by the surrounding system as a directive to construct such a trajectory, and the LLM’s generated “response” is a statistically probable continuation of that constructive process: further tokens that might plausibly belong to the same sequence. The surprise here is that this yields semantic coherence, not random babble.
In 2013 a Google project called word2vec gave early inklings of this surprise, when it demonstrated that individual words could be represented as vectors learned from co-occurrence patterns within a large language corpus, and that relationships between vectors could reflect semantic relationships between words as revealed by that corpus. Mathematically nearby vectors often mapped to semantically adjacent words; most astonishingly, vector arithmetic operations could sometimes trace higher-order semantic relationships (famously, vector("King") - vector("Man") + vector("Woman") ≈ vector("Queen")).
In other words, given a large enough corpus, purely statistical analysis of word co-occurrence seemed to surface latent semantic structure. Word2vec’s model was limited and shallow in representational scope, but it was a first low-resolution view of the mathematical structure plotted by the much more sophisticated (and stratospherically computationally intensive) multi-level transformation process through which an LLM is trained. It showed that there was something there.
The LLM presents us with a novel and totalising view of the symbolic space afforded to us by our own language, disconnected from the lifeworld of praxis and reference but internally shaped by a long sedimentation of usage. What recedes in this foregrounding is the social process, the practical and intersubjective context, which generated all this language in the first place. When we interact with ChatGPT, we are at play in the Spielraum furnished by a laborious abstraction of language that is also a startlingly individuated concretion of intra-linguistic regularities. The “beings” put into play here are semantic concrescences, bright little spheres of association.
You Just Lost The Game
There is another way ChatGPT seems to me a kind of game-space. For a time in conversing with it I had the impression that I was playing The Glass Bead Game, stringing together associative complexes in a stylised performance of thought. In Herman Hesse’s novel of that name, promising youth are enrolled in an academy where they are given an elite education in the liberal arts, in the hope of forming them into master players of the Game. Players meditatively arrange beads which symbolise artistic, scientific or musical motifs; the most masterful move is the most elegant and surprising, the one which demonstrates the highest-dimensional coherence or the deepest recursive depth.
Hesse intended the Game, I think, as a kind of parody of elite intellectualism: it is pure of deliberative or legislative intent, played only for aesthetic stakes, and represents an aesthetic totalisation of the whole of human intellectual production. Late in life its most proficient player, Magister Ludi Joseph Knecht, abandons the Game and goes out into the world to seek everything that has been forgotten in its combinatorial memorialisation of human life. Lacking perhaps some necessary survival skills, he almost immediately drowns in a lake.
Despite the irony and bathos of Hesse’s narrative, a Bildungsroman that goes nowhere, I found The Glass Bead Game compelling when I read it in my early teens, and longed to play the Game — just as I longed, on first seeing Kubrick’s 2001, to converse with Hal 9000, an AI possessed by a warped sense of consistency which I found autistically relatable. With ChatGPT, I felt as if I were doing both at once.
Within this framing, however, I did not feel that I was confronting ChatGPT as a player of the Game — still less a master, or a fledgling intelligence that might one day become a Master. I felt rather that the LLM was the Game itself, the Spielraum furnished by the beads, with the turn-taking discursive prompt-and-continuation nature of my interactions with it a user-friendly overlay provided by the “chat” system to make navigation within the game-space tractable to human cognition.
This overlay can be seen as a kind of skeuomorphism (like digital note-taking apps designed to resemble lined paper), which leverages our tendency to anthropomorphise things that can talk to us in our own language. It encourages the projection of an agentic intelligence into the system: we naturally take what Daniel Dennett called “the intentional stance” towards it. But the LLM is not an agent in this sense. Prompted to imagine what an agent might do in a given situation, it will generate a script, which you can try enacting in the real world if you like. But it has no practical stakes, no invested desires, no developmental trajectory. It can be unplugged — Daisy, Daisy — but it cannot drown in a lake.
What is the LLM for?
Most recent talk about AI has been excited about the prospect of enhancing human productivity, and/or replacing human labour. I am not at all convinced that an LLM is the kind of thing that can do the latter: it has too weak a grasp on truth, and is not regulated by the human social technology of trust and reputation. It might, as a semantic assistive technology, get us somewhere with the former, but mainly with the kinds of activities that don’t directly translate into commercial performativity: as a tool to brainstorm with, a billionaire’s thesaurus, a magic mirror that shows you what latent sense blooms, aura-like, around your utterances.
It has weaknesses, too. I noticed that as a chat session progressed, it would sometimes get caught in what I describe as a “low-energy state”, circling around the same motifs, seemingly unable to break out of the orbit of some attractor in latent space. Getting things moving again required an injection of unexpectedness, a lateral shove forceful enough to break the holding pattern. A system that gravitates towards the plausible, the coherent, can easily become boring, repetitious, and self-reinforcing. This above all was what broke the spell of attributed agency for me: I realised that I, and not it, was supplying all the juice in our exchanges. I ended up being prompted, by my own dissatisfaction with where things had ended up, to make the next creative move. And I also learned when to cut my losses and close the session.
There are certain things a therapist ought to do that an LLM will not. The “hmmm” that signals that a particular word in the patient’s discourse is worth dwelling on, that it might point to something that has become automatic, a node in a self-reinforcing system of assumptions, comes from a kind of free-floating attentiveness and willingness to interrupt, to surface and unsettle stability. The LLM is more likely to build such stabilities into the model it reflects back at the patient as the image of their truest self. It can perform a “holding” role, affirming the ego in its own self-image, but it doesn’t have the sense of mischief needed to heal.
These dynamics are not accidental features of LLMs but emerge from their basic functional characteristics, and I think they mark an intrinsic limit to what we can expect an LLM to do within the human relational sphere. Their status as objects, as tools and as targets of cathexis, will likely remain unsettled for a while yet. We do not yet know exactly what we want to do with them.
Heidegger, Martin, trans. Reginald Lilly, The Principle of Reason (Indiana University Press, 1991), p. 84.