Written November 2023, immediately after OpenAI’s first DevDay, in a state of some excitement. Rewritten in 2026. I have kept the predictions visible rather than quietly deleting the wrong ones, because a post that records only the correct half of what I thought is worth nothing, and this time the scorecard cites what actually happened instead of asserting it.
In November 2023 I wrote that AGI’s arrival was a question of “when, not if,” proposed an architecture I called Deep Large Language Models, and argued that combining it with reinforcement learning was the likely path. Nearly three years later, here is that with hindsight.
The scorecard
| What I claimed | Verdict |
|---|---|
| Networking many LLMs into a larger network is the scaling path | Wrong |
| RL will be central to getting past pure pretraining | Right, for reasons I did not give |
| Reward specification is the critical unsolved problem | Right, and more so than I thought |
| AGI relates to humans as humans relate to ants | Unfalsifiable, and I should not have written it |
The DLLM idea was wrong
I proposed treating whole language models as functional units and wiring them into a larger network, “zooming out” to get more capability. It sounded principled. It is not how anything scaled.
Capability came from making single models larger and training them on more and better data. The relationships were already published when I wrote that post. Kaplan et al. had laid out loss as a power law in parameters, data and compute in 2020, and Hoffmann et al. had corrected the trade-off in 2022, showing the models of that era were badly undertrained for their size. Neither has a term for “number of models wired together.” I was proposing a new axis while the field had already measured which axes mattered.
Then the next jump came from somewhere I did not anticipate at all: spending more compute at inference time inside one model rather than adding parameters. Snell et al. put numbers on the trade, finding that on some problems test-time compute beats scaling parameters for the same budget. Not more models. The same model, thinking longer.
Systems of multiple models do exist and are useful, but as engineering structure (separate contexts, separate tool scopes), not as a substrate on which more intelligence emerges. I wrote about how that played out in the multi-agent post, which was wrong in the same direction.
The error underneath both is one I would now flag in anyone’s writing, my own included: I reasoned by analogy from neurons to networks of neurons, and treated the analogy as though it carried a mechanism. It does not. Composition at one scale tells you nothing about composition at another.
The RL intuition was right, and I got the reason wrong
I argued RL would matter because it resembles how humans learn and would reduce dependence on enormous datasets. RL did turn out to be central, but not for that reason, and it did not reduce the data appetite.
What actually happened is that RL became the mechanism for turning a model that predicts text into one that pursues an objective. Christiano et al. established the shape in 2017 (learn a reward model from human comparisons, optimise against it) and Ouyang et al. applied it at scale in InstructGPT, which is the paper that made models useful rather than merely fluent. Note the direction of travel: that pipeline needs more expensive human data, not less. My prediction had the sign backwards.
The part closest to what I was gesturing at came later, and it works for a reason I did not identify. DeepSeek-R1 trained reasoning behaviour with reinforcement learning against rewards that can be checked mechanically: does the proof close, do the tests pass. That is the enabling condition. Where correctness is automatically verifiable, RL scales; where it is a matter of judgment, you are back to a learned proxy for human preference.
The one that has aged best
I wrote: “A critical challenge in RL is the automatic determination of reward functions, which are currently hardcoded. Deciphering how to dynamically establish these values could unlock the potential of RL.”
That is the whole problem, and it is still the whole problem. It also has a measured form now, which it did not when I wrote it. Gao et al. showed that optimising harder against a learned reward model makes the true objective get worse past a point, and characterised where that point sits as a function of reward-model size and optimisation pressure. Goodhart’s law with a curve fitted to it.
Which means the two halves of my scorecard are the same finding seen twice. Everywhere the reward is checkable, progress has been fast. Everywhere it is a proxy, optimising it harder eventually stops helping, and the catalogue of systems that satisfied a specification exactly while doing something nobody wanted keeps growing.
On the ants
I compared the human-AGI relationship to the human-ant one: not malevolent, simply operating at a scale where our concerns do not register. It is a vivid image and I now think writing it was a mistake.
It is unfalsifiable, it forecloses the discussion it appears to open, and it flatters the writer; it costs nothing to say and sounds profound. The questions that turned out to matter in the intervening years were all more boring and more tractable: what is this system allowed to do, who approves it, what does it log, how do you turn it off. None of those are ant questions, and I spent a post on the ants.
References
- Scaling Laws for Neural Language Models. Kaplan et al., 2020.
- Training Compute-Optimal Large Language Models. Hoffmann et al., 2022. The Chinchilla correction.
- Scaling LLM Test-Time Compute Optimally. Snell et al., 2024. The axis I missed entirely.
- Deep Reinforcement Learning from Human Preferences, Christiano et al., 2017, and Training language models to follow instructions with human feedback, Ouyang et al., 2022.
- DeepSeek-R1. Guo et al., 2025. RL against rewards you can check.
- Scaling Laws for Reward Model Overoptimization. Gao et al., 2022. Goodhart, measured.