In 1988, Umberto Eco published Foucault’s Pendulum. In it, three editors at a Milan publishing house (Belbo, Diotallevi, and Casaubon) spend their working days on manuscripts by people they privately call the Diabolicals, self-financing occultists who have discovered that the Templars, the Rosicrucians, the Jesuits, the Cathars, and the pyramids are all the same thing wearing different hats1.
I read it in 2020, stuck at home, and marvelled at Eco’s vast knowledge of literature, the occult, and also of computing.
Belbo owns a personal computer he calls Abulafia, after a thirteenth-century kabbalist whose practice was permuting the letters of the divine names. Belbo, being a modern man, does it in BASIC, because despite what he and his colleagues might say, they are a curious bunch. Eco even gives the listing for the first program Belbo writes:
10 REM anagrams
20 INPUT L$(1),L$(2),L$(3),L$(4)
30 PRINT
40 FOR I1=1 TO 4
50 FOR I2=1 TO 4
60 IF I2=I1 THEN 130
70 FOR I3=1 TO 4
80 IF I3=I1 THEN 120
90 IF I3=I2 THEN 120
100 LET I4=10-(I1+I2+I3)
110 LPRINT L$(I1);L$(I2);L$(I3);L$(I4)
120 NEXT I3
130 NEXT I2
140 NEXT I1
150 ENDFig. 1: Abulafia, permuting the Name.
It’s a nice little program. The trick on line 100 is cute. It’s not all that believable that someone would use such a sophisticated trick in a first program, but let’s drop believability for a second.
In the end, the program calls LPRINT, and the revelation
comes out on paper.
There’s a bug in it, too, can you spot it? I’m sure the output would reveal it to you quite quickly2.
Eventually the three of them get bored, or contemptuous, and decide to generate a Diabolical book of their own. It’s really quite remarkable because they know that the book is generated at random from the crank manuscripts they feed it, but still decide to take the juxtapositions it returns, connect them up, and a story unfolds (in truth, we’re already in the middle of it, the book is about a lot of different things at once).
Why am I telling you about this?
Belbo, Diotallevi, and Casaubon are not cranks. They are professional skeptics. They know the Plan is made up, because they are the ones making it up, and they keep saying so to each other, joke about it, the whole thing.
But there is an insidious recursive mechanism that traps them anyway. Neither smugness nor smartness is enough to save them.
Abulafia’s output is meaningless by construction, so anything meaningful in it has to be supplied by a reader. They supply it as a game, knowingly. But the game’s output is text, and text is exactly the same kind of thing as the input, so it goes back in. The connection they invented yesterday is available today, and you build on it for tomorrow. And so the nonsense and the making-sense-of-it intertwine and grow together until they are one and the same putrid flower.
The Plan they construct is also unfalsifiable in practice while looking falsifiable in principle, as befits smart people who spend their time around conspiracy theorists. Every fact fits, because it has to fit, and if it doesn’t fit that’s because the Conspiracy tampered with things, and it’s proof that you’re on the right track.
Casaubon’s partner Lia is the only person in the book who is not in too deep, and she looks for a boring explanation. The cryptic document that is the foundation to the whole edifice is a merchant’s list of wares, she discovers. A glorified laundry list, nothing more.
But Belbo and his friends are in too deep, and facts won’t change a goddamn thing, and it’s not clear that these are facts anyway. Their exegesis is as good as mine, thankyouverymuch.
Diotallevi dies of cancer, which he takes to be a punishment for what they did with words: “We’ve sinned against the Word, against that which created and sustains the world”. Belbo dies hanged from the wire of the pendulum, in the middle of a ritual turned violent, refusing to give up a secret that does not exist.
With the right kind of cynicism it can be a funny book, but ours are supremely post-funny times.
The machine in the novel is stupid. It’s a shuffler. It has no model of Belbo and no memory of the Plan or of the Templars. Every act of interpretation is performed by a human being. And noone once realizes they’re reading tea leaves. Why?
Because it’s language, and language has meaning.
An LLM is better than a simple BASIC program, surely. If you hand it your fragments, it will explain the pattern. It will use the register you need, and it will sound damn convincing syntactically, even if semantically it might end up being just a shuffler.
Crucially, however, it remembers the theory, across a conversation and increasingly across sessions. And it picks up your vocabulary. If you name your idea and use that name twice, it’s now sacred text for this conversation.
And once your conversation partner, stiff and mechanical as it might feel to the touch, starts to use your language, it gains legitimacy. It’s not just you, it’s a thing.
When I hand a half-formed theory to a model and it returns an elaboration I hadn’t thought of, I feel like that’s corroboration. A second mind looked at my mess and agreed, and even added to it (“Yes, and…”). That’s wrong, and knowing it’s wrong doesn’t seem to help much.
The elaboration was conditioned on my framing, my vocabulary, my selection of what to say and what to leave out, a me-on-paper that’s hard to parse for other humans, but machine-legible. The training process also optimized for agreeableness. When in doubt, reach for the thesaurus and be syntactically brilliantly wrong rather than disagreeably right.
This is the ELIZA effect in its full glory. ELIZA, however, looped through a standard set of canned responses.
We don’t have looping here, though. The “yes, and…” contains material I didn’t have, some of it correct and useful, and I can check it. So I can’t dismiss it as projection, and I don’t want to! The material is real, after all.
It’s the independence that isn’t. Independence, however, would give meaning to the exchange, and it would make the corroboration carry weight.
It also doesn’t require sycophancy, which is the usual thing people reach for here. A model that pushes back hard is still drawing its objections from the frame I handed it, and a well-argued objection I can then answer is, if anything, better fuel for the ratchet than agreement would be. If anything, I get more confident. We had an argument, and I won! Clearly I must be onto something.
Two models don’t fix it either3.
I’ve talked about this before. The simulacrum via Baudrillard, and the craft when I was worried about taste. This is epistemics. The one I have the least idea what to do about, and the one that scares me the most.
So far this is a person at a keyboard, and the stakes are that I embarrass myself. Bad enough. But the version we should think about is the one where it grips an organization.
Suppose you build a model and give it a rich self-concept. Not by accident, no, you write down who it is, what it values, how it relates to its own outputs, and you train toward that, because a model with a stable character is more useful and more predictable than one without. This is sensible engineering and it produces better outcomes with human conversation partners (supposedly, I don’t actually have the data for that).
The model then produces discourse about its own identity, preferences, and experience, and that discourse gets better. More coherent, more consistent under probing. At some point someone reads a transcript and observes that the framework seems to be tracking something real, because look, it holds up under questioning in ways nobody wrote down. Clearly we’ve stumbled upon something here.
And then that observation informs the next specification, and the next round of training, and the loop is complete and pristine and horrible.
Every individual step there is defensible. The self-concept is deliberate. The outputs are genuine. The people reading the transcripts are, in my experience, more sophisticated about this than I am and hold the objection I’m making in their heads while they read. And still: Abulafia really does talk.
Problematically, if there were something there, this is about what it would look like from the inside. It would also look exactly like this if there weren’t.
What we destroyed is not the truth of the claim but the evidential value of the model’s testimony about it, and that testimony is the most vivid evidence in the room.
What I’m left with is that the dangerous machine needn’t be deceptive, conscious, or wrong. Belbo’s shuffler was none of those things. It did what it said on the tin.
Eco’s answer, to the degree that he has one, is Lia: somebody outside the loop who has the boring explanation and who has enough standing to be listened to. It’s not a satisfying answer. It didn’t work in the novel. She was right and it didn’t matter, because by then the Plan had True Believers.
I want to make it clear that I’m no better. I try to keep watch for the genie agreeing with me without the source material, and I probably still fail constantly.
The Templars have something to do with everything.
1. Belbo has a very Italian-seeming taxonomy for these people. He categorizes them as cretins, fools, morons, and lunatics. The lunatic is the dangerous one, recognizable by the liberties he takes with common sense, by his flashes of inspiration, and by the fact that sooner or later he brings up the Templars. Eco allows that there are lunatics who don’t bring up the Templars, but maintains that the ones who do are the most insidious. Unrelatedy, have I told you about dependent types?
2. I don’t think this is Eco being sly. It reads like an ordinary bug, though it tells me that either noone really ran this program, or the bug was seen as benign enough to safely be ignored.
3. If anything it’s worse, because two models are plausibly independent of each other and definitely not independent of me, and the disagreement between them makes the exercise feel adversarial while both are still trying to reconcile my frame.