Written in October 2023, a few weeks after Mistral 7B was released. Rewritten in 2026, because the original had two of the architecture descriptions wrong, which is a bad thing to leave standing in a post whose whole purpose is explaining the architecture. Corrections and what actually held up are below.
Mistral 7B arrived in September 2023 and did something that had not happened before: a 7-billion-parameter model that beat Llama 2 13B across every benchmark its authors evaluated, and beat Llama 1 34B on reasoning, mathematics and code. Apache 2.0. No waitlist, no research licence, no acceptable-use policy that a lawyer had to read.
Two architectural choices carry most of that, and both are about inference cost rather than quality. That is the part I find worth understanding, because it is where the leverage turned out to be.
Sliding-window attention
Standard attention has every token attend to every previous token. Cost grows with the square of sequence length, and the KV cache grows linearly, which is what actually hurts when you are serving.
Sliding-window attention restricts each token to the W most recent positions. The obvious objection is that you have just capped the model’s memory at W tokens, and the obvious objection is wrong.
The window applies at each layer, to the layer below. So a token at layer 1 has summarised the W tokens before it; at layer 2 it reads W tokens that each already summarised their own W; and the reach compounds. After k layers, information can travel roughly W×k positions.
In Mistral 7B, W = 4096 across 32 layers, a theoretical attention span of about 131,000 tokens, from a mechanism that never computes an attention score across more than 4,096.
Correction. The original version of this post said SWA “divides the input sequence into fixed-size windows and performs attention computations within each window independently.” That is a description of blocked attention, and it is a different and worse mechanism: independent blocks cannot pass information across a block boundary at all, at any depth. The whole point of the sliding window is that the windows overlap, and the overlap is what makes the receptive field grow. I had the mechanism backwards.
Grouped-query attention
The KV cache is the thing that eats memory during generation. With 32 attention heads you store 32 sets of keys and values for every token you have generated.
GQA has several query heads share one key/value head. Multi-head is one KV per query head; multi-query is a single KV for all of them, which is fast and costs quality; GQA sits in between and picks the trade.
multi-head Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
| | | | | | | |
K1 K2 K3 K4 K5 K6 K7 K8 8 KV heads to cache
grouped-query Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
\__/ \__/ \__/ \__/
K1 K2 K3 K4 4 KV heads to cache
multi-query Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
\_____\_____/_____/
K1 1 KV head to cacheCorrection. The original said GQA “groups queries together based on their similarity” to remove redundant computation. It does not. The grouping is fixed by the architecture before training and has nothing to do with how similar any two queries are at runtime: there is no similarity computation and no dynamic clustering. Query heads stay fully independent; only the keys and values are shared.
What the benchmarks actually said
Correction, and this is the one that mattered most. The original post claimed Mistral 7B “consistently outperforms other models by a significant margin” on BLEU, and used that to argue it generates more accurate translations.
The Mistral 7B paper does not report BLEU. Translation is not among the evaluated tasks. I do not know where that claim came from. Most likely I absorbed it from one of the secondary write-ups the original post cited instead of reading the paper, which is exactly the failure mode a post like this exists to avoid.
What the paper does claim, and what held up:
- Outperforms Llama 2 13B on all evaluated benchmarks.
- Outperforms Llama 1 34B on reasoning, mathematics and code generation.
- Mistral 7B Instruct beats Llama 2 13B Chat on both human and automated evaluation.
Note what is not in that list: nothing about translation, and nothing about being state of the art in absolute terms. The claim is about punching above its parameter count, and that claim was correct.
The licence was the story
It is easy to forget how much this mattered in late 2023. Llama 2 came with a custom licence and an acceptable-use policy. Mistral 7B came under Apache 2.0, with the weights on a torrent link.
That meant you could fine-tune it, ship it in a commercial product, run it on a customer’s own hardware, and not have a conversation with legal first. A very large amount of the open-weights tooling that exists today got built against this model, for that reason rather than for any architectural one.
What held up, from 2026
Both mechanisms are now ordinary. GQA is in essentially every open-weights model of consequence; it turned out to be a free lunch and everyone took it. Sliding-window attention had a more mixed run: it is genuinely useful, but the “theoretical span” framing oversells it, because information degrades as it propagates up through layers and a 131,000-token theoretical reach is not 131,000 tokens of usable context. Later long-context work leaned more on position-encoding changes and on hybrid designs that interleave a few full-attention layers among the windowed ones.
The durable lesson is not either mechanism. It is that in late 2023 the binding constraint on open models stopped being quality and started being the cost of serving them, and the team that optimised hardest for inference cost got the ecosystem.
References
- Mistral 7B. Jiang et al., the paper. Nine pages, worth reading directly rather than in summary, as I should have done the first time.
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. Ainslie et al., where grouped-query attention comes from.
- Longformer and Generating Long Sequences with Sparse Transformers. The sliding-window lineage Mistral draws on.