Written September 2023, rewritten in 2026. The original answered “do language models remember?” with “no, they only recognise patterns; they don’t store specifics.” That was wrong, and it was wrong in the direction that lets people put things into a model they should not. The corrected version is below.
“Do large language models remember?” is three questions wearing a trench coat, and they have different answers.
Ask an engineer and they hear: does it recall what I said earlier in this conversation? Ask a lawyer and they hear: could it reproduce the document we trained it on? Ask a philosopher and they hear something about experience. The first two have precise, measurable answers that point in opposite directions, and conflating them is how organisations end up training on data they should not have.
Yes, models memorise. This is measurable.
The claim in the original version of this post (that models “generate outputs reminiscent of their training” but “don’t store specifics”) had already been falsified when I wrote it.
Carlini and colleagues demonstrated in 2021 that you can extract verbatim training sequences from GPT-2, including personally identifiable information, by generating text and checking which continuations the model is unusually confident about. In 2022 they quantified it, and found three log-linear relationships. Memorisation grows with:
- model scale: bigger models memorise more of their training data, not less;
- duplication: the more times a sequence appears in the corpus, the more likely it is recoverable;
- context length: the more of a sequence you feed back, the more reliably the model completes the rest of it.
All three run the wrong way for anyone hoping this problem shrinks on its own. Scale is the direction the field is going, near-duplicates are endemic in web corpora, and context windows have grown by orders of magnitude since that paper.
In 2023 a follow-up showed the same thing on a deployed product: asking ChatGPT to repeat a single word forever eventually caused it to diverge from the repetition and emit chunks of training data. The mitigation for that specific attack was a filter on the specific behaviour, which tells you where the data was: still in the weights.
A language model is a lossy compression of its training corpus. Lossy compression is not the same as not storing it. JPEGs are lossy and you can still recognise the face.
And no, it does not remember your conversation
The opposite error is just as common and produces the opposite bad decision. People assume that because a model “remembered” what they said ten messages ago, it has learned it, and will carry it forward.
It has not. What happened is that the entire conversation was resent with the request. The apparent memory is the transcript being re-read from scratch every single turn, at a cost that grows with the length of the conversation. Nothing was written to the weights. Close the session and it is gone.
This is why a model can be word-perfect on something you said an hour ago and blank on something from last week, and why the fix for “it keeps forgetting” is never fine-tuning. It is a retrieval system.
Three memories, three sets of rules
Which is why the diagram at the top is the useful way to hold this. The three things all called memory have different persistence, different editability, and completely different disclosure risk.
| Weights | Context | Retrieval | |
|---|---|---|---|
| Written when | training | each request | when you index |
| Persists | forever | one request | as long as the store |
| Can you delete an item? | no, only retrain | yes, trivially | yes, it’s a database |
| Can it leak training data? | yes | no | only what you indexed |
| Auditable | barely | yes | yes |
The row that should drive decisions is the fourth from the top. “Can you delete an item?” is a regulatory question in most jurisdictions I would want to operate in, and for weights the answer is no. There is research on machine unlearning; none of it is a delete statement.
The practical consequence is a rule I would apply without much argument: anything you might one day need to delete does not go in the weights. Put it in a store the model reads from. You give up nothing except a small amount of latency, and you keep the ability to answer a deletion request without retraining a model.
The part where the original post was hand-waving
It also claimed “the AI world has robust measures to ensure data used for training doesn’t have personally identifiable information.”
There are measures. Deduplication genuinely reduces memorisation, and it is the highest-leverage single intervention because duplication is one of the three factors above. PII scrubbing exists. Differential privacy in training exists and costs accuracy. Output filters exist and are the weakest layer, since they only catch what they were built to catch.
But “robust” was doing a lot of work in that sentence, and it was not earned. What is true is that these are mitigations with measurable, partial effects, applied unevenly, on corpora nobody has fully audited. Writing that as a reassurance was the least defensible thing in the original post.
So: do they remember?
The weights remember, imperfectly and permanently, in a way you cannot inspect directly or edit surgically, and more so the larger they get. The context does not remember at all; it re-reads. Retrieval remembers exactly what you told it to and forgets what you delete.
Only the first one is memory in the sense the question usually intends. It is also the only one you cannot take back, which is why it is the one worth being careful about.
References
- Extracting Training Data from Large Language Models. Carlini et al., 2021. The one that started it.
- Quantifying Memorization Across Neural Language Models. Carlini et al., 2022. The three scaling relationships.
- Scalable Extraction of Training Data from (Production) Language Models. Nasr et al., 2023. The repeat-a-word divergence attack.
- Deduplicating Training Data Makes Language Models Better. Lee et al. Why deduplication is the highest-leverage mitigation.
Debasish Sarkar
Very nicely explained.