Written October 2024, rewritten in 2026. The original proposed six steps toward curiosity-driven AI without citing any of the twenty years of work on it, and one of the six was blockchain. This version keeps the premise, drops the blockchain, and starts from the result that the field already had.
Curiosity as an engineering idea is simple and old. Give the agent a reward for encountering things it does not understand, and it will go looking for them without you having to specify what to look for. Schmidhuber formalised a version of this in 1991.
It also contains a trap that is worth understanding before you build anything, because the obvious implementation walks straight into it.
The obvious implementation
If you want an agent that seeks out what it does not understand, the natural move is to build a model that predicts what happens next and reward the agent in proportion to how wrong that model was. High prediction error means the agent has found something it cannot yet explain. That is Pathak et al.’s intrinsic curiosity module, roughly, and it works impressively well on Mario and VizDoom.
Now put a television in the world.
The corridor below has 25 rooms. Movement is deterministic and the rooms all look the same each time you visit them, except room 5, which shows static. It looks different every single time, at random. The rooms worth finding are all behind it.
class Corridor:
def step(self, a):
self.s = clamp(self.s + (1 if a else -1))
return self.s
def observe(self, s):
"""What the room looks like. The TV never looks the same twice."""
if s == TV_ROOM:
return self.rng.uniform(0, N_ROOMS) # static on the screen
return float(s)A prediction-error agent finds room 5 quickly, and its forward model starts trying to predict what the television will show next. It cannot. The error stays high forever, so the intrinsic reward stays high forever, so it stays.
| Time at the TV | Rooms found | |
|---|---|---|
| Prediction error | 46.2% | 9.7 / 25 |
| Novelty (visit counts) | 5.3% | 21.9 / 25 |
Mean of 20 runs, 20,000 steps each. The prediction-error agent spends nearly half its life watching static and never gets past room 9. Look at the shape of its curve in the figure: it explores faster than the novelty agent for the first 5,000 steps, then flatlines completely. It did not fail to explore. It succeeded, found the most interesting thing in the world by its own definition, and stopped.
This is the noisy-TV problem, and it is not an implementation bug. The agent is maximising exactly what you asked it to maximise. You asked for the wrong thing.
The fix in that experiment is a different definition. Reward novelty (one over the square root of how many times you have been here) and the TV stops being special, because it is a state like any other and its count goes up like any other. That agent reaches 21.9 rooms.
What the field actually did about it
Counts do not scale: in any real environment you never visit the same state twice, so every count is one. The real solutions are more interesting than my toy.
- Inverse dynamics features (Pathak et al., 2017). Do not predict the raw observation; predict in a feature space trained to recover the action that connected two states. Anything the agent cannot influence, like television static, is useless for predicting actions and gets squeezed out of the representation. Elegant, and it handles a large class of distractors.
- Random network distillation (Burda et al., 2018). Predict the output of a fixed, randomly initialised network applied to the current state. The target is deterministic given the state, so error goes to zero on anything visited often, however unpredictable its dynamics. This is the most-used answer in practice, and it is a genuinely clever sidestep: it converts “is this predictable?” into “have I seen this?”
- Go-Explore (Ecoffet et al.). Stop trying to be clever about intrinsic reward and instead remember promising states, return to them deterministically, and explore from there. It beat Montezuma’s Revenge, which every curiosity method had been failing at, largely by noticing that the hard part was getting back to the frontier.
The large-scale study by Burda et al. is the one I would read first if I were starting now. They ran purely curiosity-driven agents, with no external reward at all, across 54 environments, and the results are both better and stranger than you would guess.
What this means for the data bottleneck
The original framing of this post was that AI is limited by human-curated data, and curiosity is how a system generates its own. I still think that is the right problem. The noisy-TV result sharpens what a solution has to do.
An agent that generates its own training data is choosing its own curriculum, and the choice of what counts as interesting is the curriculum. Get it wrong and the system does not fail loudly; it works enthusiastically on the wrong thing, which looks like progress from the outside. Half of the prediction-error agent’s steps were spent on high-reward, perfectly optimised, completely worthless exploration.
That failure mode transfers well beyond reinforcement learning. Any system that selects its own next task against a learned proxy for “interesting” can find its own television, and the more capable it is, the faster it will find one.
Corrections to the original
- The blockchain step is gone. The original proposed decentralised AI networks with blockchain ensuring the integrity of shared training data. There is no problem in curiosity-driven learning that this addresses, and I included it because it was 2024 and it sounded like the future.
- The citations are new. A post about curiosity-driven learning that cites nobody working on curiosity-driven learning was the core failure, and it is why the original could propose six steps without noticing that step one has a known counterexample.
- “Six steps” is gone. It implied a roadmap I had not walked. What I have instead is one experiment and a reading list.
References
- Curiosity-driven Exploration by Self-supervised Prediction. Pathak et al., 2017. The intrinsic curiosity module and the inverse-dynamics feature space.
- Large-Scale Study of Curiosity-Driven Learning. Burda et al., 2018. 54 environments, no external reward, and the noisy-TV experiment done properly.
- Exploration by Random Network Distillation. Burda et al., 2018.
- Go-Explore. Ecoffet et al. Remember the frontier and return to it.