Written November 2024, rewritten in 2026. The original proposed five capabilities AGI would need and gave each an “experiment idea” I never ran. Proposing experiments is cheap. This version keeps the one argument I still think is right, is explicit about which parts were speculation, and, unlike the original, cites the work it is standing on.
The standard picture of AGI is a better version of us: same kind of mind, more of it. I think that picture is doing damage, not because it is necessarily wrong about what we will build, but because it decides in advance what we will accept as evidence.
The Turing test measures the wrong thing, and Turing knew it
It is worth reading the 1950 paper rather than the folklore version. Turing does not propose the imitation game as a definition of thinking. He opens by considering the question “can machines think?”, calls it too meaningless to deserve discussion, and replaces it with a game because the game is operational and the question is not.
That is a much more modest claim than the one the test is usually cited for, and the modesty is the interesting part. He substituted a measurable proxy for an unmeasurable property, which is exactly what benchmarks do, and it inherits exactly the problem benchmarks have: the proxy and the property come apart under optimisation pressure.
Systems now pass casual versions of the game routinely, and the response has not been to declare the problem solved. It has been to notice that indistinguishability was never what we cared about. Imitation is a test of a system’s ability to model us. A system could model us perfectly and understand nothing else; a system could understand a great deal and be obviously non-human in conversation. The test is orthogonal to the property. The Stanford Encyclopedia entry is the best short survey of the sixty years of objections.
The one argument I still believe
Human cognition is a particular solution to a particular set of constraints: a specific body, a specific lifespan, a working memory of about seven items, a visual system tuned to a narrow band of the spectrum, and a strong prior that time runs forwards and objects persist.
None of those constraints apply to a system trained on text. It has no body, effectively no lifespan, a working memory bounded by a context window rather than by rehearsal, and it learned about the world from a compressed description of it rather than from being in it.
Expecting that to converge on human-shaped cognition is like expecting convergent evolution between a bird and a helicopter because both solve flight. Both fly. Only one of them can hover indefinitely, and only one can land in a tree.
The practical form of this is that a system’s failures will be as alien as its capabilities. The characteristic modern failure (fluent, confident, wrong, on something a child would get right, while simultaneously being useful on something most adults cannot do) is not a bug on the way to being fixed. It is what a differently-shaped competence looks like from the inside of a human evaluation.
Measured, rather than asserted
That paragraph was the whole thesis and I had never checked it, so here it is as an experiment. Eight task families, ordered by how hard they are for a person. That ordering is my judgement and it is the independent variable, so it is printed on the figure for you to disagree with. Twelve freshly generated instances of each, so nothing depends on a benchmark that might be in the training data, and every answer is checked against a value Python computed rather than by asking a model to grade a model.
If machine competence were shaped like human competence, those bars would get shorter as you go down. They do not.
| Tasks a child does | Tasks needing training | |
|---|---|---|
| gpt-4o-mini | 58% | 88% |
| gpt-4o | 92% | 79% |
The smaller model is thirty points better at the tasks a person needs training for than at the tasks a seven-year-old does perfectly. It writes a word backwards correctly four times in twelve, and computes a modular inverse correctly twelve times in twelve. One of those is a party trick for a child and the other is a thing most adults have never been taught.
The sharpest single point is not at either end. Multiplying two three-digit numbers, a task any adult can do with a pencil and no special knowledge, is the worst task for both models, at 8% and 58%. It sits between the child tasks and the specialist tasks on the human scale and below both on the machine scale. There is no way to redraw a single difficulty axis that puts these in order.
Scale changes the picture without fixing it. Going from mini to gpt-4o lifts the child-level tasks from 58% to 92%, which is what you would expect if the jaggedness were a transient artefact of small models. But the specialist tasks go down, 88% to 79%, and three-digit multiplication stays broken. The curve got higher; it did not get human-shaped.
What this does not show. Two models from one family, twelve trials per cell, no reasoning models. A system that spends inference compute would paper over several of these, and that is worth measuring separately, because “we fixed it by thinking longer” is a different claim from “the competence was human-shaped all along.” My difficulty ordering is also a judgement, not data. The result I would defend is narrow: on these eight tasks the two orderings disagree, and they disagree most in the middle.
The original post reached for Mary’s Room here to argue that knowledge without experience is incomplete. I have cut that. Jackson’s knowledge argument is about phenomenal consciousness, and I was using it as though it were about generalisation, which are different claims. Borrowing a thought experiment because it gestures in a congenial direction is how philosophy gets misused by engineers, and I was doing it.
What I would retract
“Self-directed evolution”: a system rewriting its own architecture on the fly. I described this as though the obstacle were ambition. The obstacle is arithmetic: you cannot reliably evaluate an architecture without training it, so a system rewriting itself has no cheap way to know whether it improved. That is not a guess. Neural architecture search with reinforcement learning did roughly this, deliberately, and the original experiments consumed thousands of GPU-days to search a space far narrower than “any architecture.” The subsequent decade of that field is mostly about making the evaluation cheaper, which is the actual bottleneck I waved past.
“Ethical self-alignment”: a system deriving its own morality through multi-agent simulation. This assumes the hard part of alignment is that we have not asked nicely. The hard part is specification: we cannot write down what we want precisely enough to check whether a system has it. Concrete Problems in AI Safety laid out the failure modes in 2016 (negative side effects, reward hacking, unsafe exploration), and DeepMind keeps a running list of agents that satisfied a specification exactly and did something nobody wanted. A system inferring ethics from a simulation inherits every flaw in the simulation, and now the flaw is load-bearing.
The experiment ideas. “Put an AGI in a universe with randomised physical laws” is not an experiment, it is a premise. I would not include a suggestion like that again without at least describing what the measurement would be.
What would change my mind
Since the original had no falsifiable content at all, here is the version with some. I would take the alien-intelligence framing much more seriously given a result that is checkable but not reachable (a proof, a construction, a design) that no human produced, and that human experts shown it can verify but cannot explain how one would have arrived at it.
Part of this has already happened, and being specific about which part is the useful move.
FunSearch (Romera-Paredes et al., Nature, 2024) produced a new construction for the cap set problem, a genuinely open question in extremal combinatorics, by searching over programs with a language model and keeping what a verifier accepted. The output is a program mathematicians can read and check, and it beat the best known construction. That is the “checkable” half, unambiguously.
What it is not is the other half. The problem was chosen by people, formalised by people, and handed to the system with a scoring function attached. The search is creative; the framing is not. Every instance of this I know of has the same shape, and it is the shape my original post assumed away.
So the version of the claim I would now defend: a system that selects its own problem, formalises it well enough to be checked, and produces something a field adopts, would be the thing. Nothing in the FunSearch line of work is close to that yet, and it is not obvious that scaling the search gets you there, because the missing capability is upstream of the search.
References
- Computing Machinery and Intelligence. A. M. Turing, Mind LIX(236), 1950, doi:10.1093/mind/LIX.236.433. Linked to a readable copy because the publisher blocks direct access. Shorter and stranger than its reputation.
- The Turing Test. Stanford Encyclopedia of Philosophy. The objections, organised.
- Neural Architecture Search with Reinforcement Learning. Zoph and Le, 2017. What it actually costs to let a system redesign itself.
- Concrete Problems in AI Safety. Amodei et al., 2016. Still the clearest statement of why specification is the hard part.
- Mathematical discoveries from program search with large language models. Romera-Paredes et al., Nature, 2024. FunSearch.