In June 2024 the EuroHPC Joint Undertaking announced that 800,000 computational hours on MareNostrum 5 would go to the fifth-ranked proposal in its Large AI Grand Challenge. That was us. The company’s announcement said the model would have a minimum of 30 billion parameters, be trained from scratch, and be done inside twelve months.
At the same time, the thing Multiverse actually sold was making other people’s models smaller.
I spent eighteen months at that company working both sides of that sentence, and the tension in it is the most interesting thing I have to say about the experience. If large language models are so overparameterised that you can throw away 70% of the weights and lose two points of accuracy, which is roughly what our compression paper reports, then why is anyone spending a supercomputer allocation training a big one from scratch?
I think the answer is more interesting than “both are true, at different times.” But it took me a while to get there.
What compression taught us about where a transformer keeps things
The part of the CompactifAI work I find most durable is not the compression ratio. It is the layer sensitivity profile, which is in the supplementary material and which almost nobody quotes.
The paper profiles LlaMA-2 7B block by block (32 attention blocks, each with four attention layers and three MLP layers), asking how far each could be truncated before the model noticed. The finding, quoting it directly:
the initial layers and attention blocks which are located at the beginning of the attention network are more sensitive to truncation … as we move towards the end of the network, the sensitivity decreases and we can compress the layers down to 10% of the original size without significant loss of accuracy.
Two practical rules fell out of that, and they are the kind of thing you only get by running the experiment: do not take early layers below 50%, and exclude the last MLP layer of each attention block from tensorisation entirely.
Read that as a statement about architecture rather than about compression and it says something uncomfortable. The back two-thirds of a 7B transformer is doing work that survives having 90% of its parameters removed. Whatever is happening in the early layers is not.
The benchmark that did not cooperate
The headline is a 93% memory reduction and a 70% parameter reduction for a 2–3% accuracy drop. I am an author on it and I still think that number needs its asterisk shown, because it is an average across five benchmarks and one of them behaves nothing like the others.
MMLU, HellaSwag, BoolQ and TriviaQA all land between 3.7% and 5.0% relative. GSM8K goes from 23.05 to 17.74: 5.3 points absolute, 23% relative, five times the next-worst result.
Read the 2–3% claim as absolute points and TriviaQA (0.93) and MMLU (2.25) fit, HellaSwag (4.01) and BoolQ (2.99) are marginal, GSM8K (5.31) is not close. Read it as relative and nothing fits especially well, and GSM8K is in a different category entirely. Either way, arithmetic is where this method costs you.
That is not surprising once you say it out loud. Multi-step arithmetic has no redundancy to spare: a chain of reasoning fails completely if any link is wrong, where a factual recall task degrades gracefully. Compression removes exactly the kind of low-magnitude precision a carry operation depends on.
So if you are compressing a model that has to do maths, or anything with a long chain of dependent steps, benchmark that specifically. The average will tell you it is fine.
Healing gets three sentences and most of the months
The paper describes the decomposition precisely: sequential SVDs on each weight matrix, keep the largest χ singular values, replace the matrix with a matrix product operator. That is three sentences of maths and it is exactly right.
Then there is the retraining step, which the paper calls healing, and which gets this:
our method also includes a rapid retraining phase, dubbed as healing, following the determination of the truncated MPOs.
Generic chat datasets: Ultrachat, Alpaca, OpenHermes. Eight A10g GPUs. Less than a single epoch. No token counts, no learning rates, no wall-clock.
The ratio of page space to engineering effort there is inverted, and I say that as someone who has since reimplemented the decomposition from scratch on GPT-2 to check my own intuitions. The decomposition is an afternoon. Getting the healing to converge, on a model that a truncation has just badly damaged, without simply overfitting whatever corpus you handed it, is the work.
This is a general property of compression papers rather than a criticism of ours. The factorisation is legible and citable, so it gets the space. The retraining is fiddly and unglamorous and mostly consists of things that did not work, so it gets a subordinate clause. If you are evaluating a compression technique, the first question worth asking is how much retraining is hiding inside the number.
So I ran it
Saying the healing is where the work is, is an assertion. Here is the version with numbers behind it: the MPO decomposition reimplemented from scratch, applied to every MLP matrix in GPT-2 small, evaluated on a held-out slice of WikiText-2. Two hundred lines, no dependency beyond torch, runs on a laptop.
The first thing I checked was not the model at all. Take one MLP weight matrix, decompose it at a range of bond ranks, and measure the reconstruction error. Then do exactly the same thing to a matrix of Gaussian noise with the same shape and standard deviation, as a control.
The two curves sit on top of each other. Across every rank I tested they never differ by more than 0.005. Under this factorisation, a trained GPT-2 MLP matrix is no more compressible than noise of the same shape.
There is a second, more practical number in that panel. An MPO with two cores costs r × (o₁i₁ + o₂i₂) parameters, which for this matrix is 3,200 per unit of rank against 2.36M dense. Parameter break-even lands at rank 737, and at rank 737 the reconstruction error is still 0.31. Every rank that saves you parameters costs you a third of the matrix or more.
The behavioural cost follows. Replacing all 24 MLP matrices and measuring perplexity, with no retraining:
| Bond rank | Parameters kept | Perplexity | vs baseline |
|---|---|---|---|
| dense | 100% | 37.7 | n/a |
| 640 | 94.0% | 231 | 6.1× |
| 512 | 86.1% | 1,271 | 33.7× |
| 384 | 78.2% | 3,950 | 104.8× |
| 256 | 70.3% | 7,065 | 187.5× |
Keep 94% of the parameters (a 6% saving, which is not a compression method, it is a rounding error) and the model is already six times worse. At the 70% figure the paper headlines, this model is producing noise.
Then heal it. Freeze everything except the cores, 800 steps of AdamW at 1e-4 on a slice of the corpus disjoint from the evaluation slice, six and a half minutes on an M4 Pro. At rank 512 that moves perplexity from 1,208 to 641: 48% of the gap back to baseline, from a retraining budget that is a rounding error against eight A10gs.
The factorisation on its own does nothing here. Everything that makes the method work is in the three sentences about healing.
Which is the point I was making above, now with a number attached rather than a recollection. It also reframes what the decomposition is for. It is not a good approximation of the weight matrix; it is demonstrably no better than what you would get from noise. It is a smaller parameter space, shaped like the original operator, for gradient descent to re-find the function in. Judging it by reconstruction error is judging it by a quantity nobody is optimising.
What this does not show
GPT-2 small is not LLaMA-2 7B, and this is the caveat that matters. Overparameterisation is a function of scale, and a 124M model trained to convergence on far less data than a 7B has much less slack to give up. Nothing here contradicts the paper’s numbers, and I would expect the same experiment on a 7B to look considerably kinder. What it does establish is the shape of the thing: that the reconstruction error is not the quantity to watch, and that the retraining is carrying the result.
Also: MLP matrices only, one seed, perplexity rather than downstream benchmarks, and a healing budget three orders of magnitude below the paper’s. A longer heal would recover more, which is rather the point.
Code, results and the figure generator: acebot712/blog-experiments, under compactifai/.
Physicists doing machine learning
Something that does not show up in a results table: this was a company where a large share of the technical staff came from condensed matter physics rather than from ML. The affiliations on that paper include the Donostia International Physics Center, Ikerbasque, and the Catalan Institute of Nanoscience. The paper is typeset in a physics style, not a NeurIPS one.
That shows up in the vocabulary (bond dimension, simple update, matrix product operator are all borrowed intact from tensor-network condensed matter), and it shows up in what people find obvious. Truncating a matrix by discarding small singular values is completely natural if you have spent years doing DMRG. Coming from ML, the natural framing is low-rank adaptation, and you have to be told that interleaving the indices before you factorise is what makes it an operator decomposition rather than just a low-rank approximation.
Both groups are right and they are describing the same linear algebra. It took longer than it should have for me to notice that most of our disagreements were vocabulary.
So why train a 30B at all?
Back to the tension. Here is where I landed.
Compression tells you that a trained model contains a great deal of redundancy. It does not tell you that you could have trained a smaller model to the same quality directly. Those are different claims, and the second one is much stronger and much less supported. The overparameterisation appears to be load-bearing during training and disposable afterwards, which is a strange property and not one anybody has fully explained.
Which means a compression company still needs someone upstream training big models, and if the frontier labs stop publishing weights you had better be able to train your own. That is a commercial argument as much as a scientific one, and it is a reasonable answer to “why both.”
The more interesting version of the question is whether the two directions ever meet: whether you can use tensor-network structure during pretraining rather than applying it to a finished checkpoint. If overparameterisation is what makes optimisation work, then constraining the parameterisation from step zero should hurt, and the interesting question is by how much and whether the healing intuition transfers. I do not think that question is settled, and it is the one I would work on if I went back.
What I am not going to tell you
Being explicit, because a post about a supercomputer allocation that is vague in the middle reads like a post by someone who was not there.
- The outcome of the 30B project is not public. There has been no follow-up announcement, and there is no 30B model on the company’s Hugging Face organisation. I am not going to be the source for that, and you should draw no inference from my not saying: supercomputer allocations produce outcomes on their own schedule and mine is not the only opinion involved.
- Nothing about internal structure: team size, who did what, why anyone left.
- No unpublished numbers. Everything quantitative above is from the paper or from a public announcement, both linked.
One correction while I am here, because I have seen it repeated: Multiverse did not win the Large AI Grand Challenge. It placed fifth and received a bonus compute allocation. Those are different things and the second one is still a good outcome.
The thing I would tell someone joining a compression team
Your intuitions about what matters will be wrong for about three months, and they will be wrong in a specific direction: you will overvalue the factorisation and undervalue the evaluation.
It is very easy to build a compressed model that looks fine. Perplexity moves a little, three benchmarks hold, and the thing generates fluent text. Then somebody asks it to do arithmetic and you discover you removed something specific. The GSM8K row is the version of that I can point at in public; it is not the only one I saw.
So build the evaluation before the method. Not as a discipline exercise, but because the method is genuinely easy and the evaluation is genuinely where the surprises are, and the ratio is not close.