The Bottlenecks of AGI: Why Can’t Machines Think Like Us… Yet?

Written September 2023 as a listicle of six bottlenecks and seven speculative solutions, one of which was quantum computing. Rewritten in 2026 to check which bottlenecks were real. Three of the six moved substantially. The ones that did not are the interesting part.

In 2023 I listed six things standing between the systems we had and general intelligence. It is a useful list to revisit, because a bottleneck that dissolves in three years was never a bottleneck; it was a limitation of the current implementation, and telling those two apart is most of the skill in this kind of forecasting.

The six, scored

Bottleneck, as I framed it in 2023	Now
Narrow domain competence	Largely dissolved
Learning stops after training	Partly worked around
Needs enormous data to learn a concept	Dissolved in-context
No goals of its own	Still true, and now deliberate
No emotional understanding	Was the wrong frame
Safety and control	Harder than I described

The three that moved

Domain narrowness was the clearest miss. I wrote that models “excel in their silo and flounder outside it,” using AlphaGo and GPT-3 as the examples. Generality across text-shaped tasks arrived largely as a side effect of scale, which is not what I expected and not what the framing predicted.

Data gluttony I described as needing thousands of examples to learn a concept. In-context learning makes this false at inference time: a handful of examples in the prompt, sometimes one, sometimes a description and none. The gluttony moved to pretraining, where it is enormous and paid once.

Static learning is the interesting middle case. Weights still freeze after training, exactly as I described. But retrieval, long contexts and tool use route around it well enough that the limitation stopped being the thing you notice, which is a different situation from it being solved. See the post on memory for why the distinction matters.

The one I framed wrongly

“The emotional abyss”: models cannot feel, only simulate. Still true and it turned out to be the wrong axis. The practical question was never whether a system feels anything. It is whether people respond to it as though it does, and they emphatically do. That produces real consequences (attachment, disclosure, misplaced trust), none of which depend on resolving the philosophical question, and all of which needed engineering attention while I was writing about qualia.

The one that got worse

I framed safety as “how do you ensure the safety of an entity that can outthink you,” which is a question about superintelligence and therefore comfortably far away.

The safety problems that actually arrived needed no superintelligence at all. A model with tool access, a plausible instruction from an untrusted source, and no boundary between the two will do something expensive, and it will do it while being much dumber than the person it harms. I ended up writing a permission matrix about this, which is roughly as far from “an entity that can outthink you” as a security control can get.

On the seven speculative solutions

The original ended with a brainstorm: digital kindergartens, neural evolution, dreaming machines, emotion as vectors, quantum computing. I labelled it “wild, witty and whimsically technical” and invited readers to code them up.

Two of those seven have real research programmes behind them that I did not cite: multimodal training was already well underway, and replay during downtime has a genuine literature in both neuroscience and continual learning. Presenting them as my whimsy rather than as active fields was the failure: it made an existing body of work look like an idea I had in the shower, which is unfair to the people doing it and useless to a reader who wanted to follow up.

Quantum computing was not on the critical path in 2023 and is not now.