Part 1: a field guide
A language model is, for the purposes of this article, two landscapes.
The first is the landscape of numbers. A model is billions of weights, and if you draw one block of them as a height map you get mostly gentle, rolling land with a few needle-sharp spires: the outliers, a handful of values hundreds of times larger than their neighbours. Compressing a model means cutting that land into terraces. Each terrace is one of a small set of allowed heights, and every weight is moved to the nearest one. Fewer terraces means fewer bits per weight.
The second is the landscape of meaning. Internally the model represents each word, and each piece of context, as a point in a space of a few thousand dimensions, where related ideas sit near each other. Compression shakes that landscape a little. Most of the time the points stay in their valleys. The harm happens when a point slips over a ridge into a neighbouring valley, which is what a different next word looks like from the inside.
We terrace the first landscape, then check whether that shook the second.
Why terrace at all
Generating text happens one token (a word or a piece of one) at a time, and for each token the model reads every weight it needs from memory. On a laptop answering one person, the time goes into that reading. The chip can multiply far faster than memory can deliver the numbers. Fewer bits per weight means a smaller model and a faster one: half the bits, half the bytes to move per token.
The two phases of a request have different limits. Reading the prompt (“prefill”) processes thousands of tokens at once, reuses each weight many times, and is mostly limited by arithmetic. Writing the answer (“decode”) processes one token at a time and is mostly limited by memory. Quantization helps decode most.
The saving starts with the number of allowed heights. Four bits give 16 terrace levels; eight bits give 256. The levels are not set once for the whole model. Weights are grouped into small plots, 32 or 64 values, and each plot gets its own spacing, a “scale” stored beside it. A plot of small weights gets fine terraces; a plot with a large value gets coarse ones. The format MXFP4 uses plots of 32 with a power-of-two spacing; NVFP4 uses plots of 16 with a finer-grained scale (an 8-bit float per block, plus one scale per tensor) (arXiv 2310.10537; NVIDIA’s NVFP4 post).
Outliers are the enemy of this scheme. One spire in a plot forces a wide spacing on every weight in it, and the other 31 values lose most of their resolution.
Terraces need not be evenly spaced either. Integer levels (INT4, INT8) are evenly spaced, simple and fast. Floating-point levels (FP8, FP4) are dense near zero and sparse far out, which suits bell-shaped weights. Other schemes place the levels where the weights actually are: NF4 (QLoRA, arXiv 2305.14314) assumes a normal distribution, and Lloyd-Max fits the levels to data. Terracing can also work across several dimensions at once. Instead of rounding each weight alone, it rounds groups of weights to points of a lattice, like packing oranges instead of stacking cubes: an E8-lattice codebook in QuIP# (arXiv 2402.04396), trellis codes in its successor QTIP (2406.11235), and learned additive codebooks in AQLM (2401.06118).
Walking around the mountain
A spire looks like a spire from one direction. Multiply a block of weights by a random rotation, and the same land, seen from another angle, becomes rolling hills: the outlier’s energy is spread over many coordinates. The rotation is undone exactly afterwards (or folded into the neighbouring computation), so nothing is lost by rotating; only the terracing in between becomes easier. Hadamard rotations are the cheap, structured version used by QuaRot (arXiv 2404.00456); SpinQuant (2405.16406) learns its main rotations instead and keeps fixed Hadamard rotations for the rest.
The three-dimensional picture cannot explain why almost any angle works. In thousands of dimensions, random directions are nearly orthogonal and a random rotation spreads any single spike almost evenly; “concentration of measure” makes the typical case very typical.
Walking around can also hurt. On floating-point 4-bit grids a random rotation helps little or even hurts, because trained weights already suit such a grid better than a rotation leaves them (KBBQ, arXiv 2609.08135, on four models trained without quantization). A model trained on a 4-bit grid is the extreme case: rotating it moves every weight off that grid, and re-terracing then costs precision the original grid did not.
Who decides where the bits go
Choosing a grid still leaves the question of where the bits should go. Calibration methods look at real inputs first. GPTQ compensates each rounding decision by adjusting the weights not yet rounded; AWQ protects the weight channels that see large activations (arXiv 2210.17323, 2306.00978). More recently, G2PTQ (arXiv 2609.31009) refreshes that compensation block by block against the model’s final output, at about five times GPTQ’s calibration time on an 8B model. Mixed precision gives some layers more bits than others.
A statistic that correlates with damage (for example a heavy-tailed weight distribution) is not a map of where repair helps; the direct way is to perturb one part at a time, measure, and repeat, since the parts interact (Dettmers, “Runtime Dynamic Compression of Mixture of Experts”, 2026-09-22).
All of this works on a finished model. Builders now often do the terracing themselves. Quantization-aware training keeps weights near the grid while the model learns, so the released 4-bit model is the model, not an approximation of it: gpt-oss ships its experts in MXFP4; Google ships Gemma 4 checkpoints trained for 4-bit deployment (2026-06-05, verified in our research brief). Some go below 4 bits, to 1 to 2 bits and ternary weights: PrismML’s Ternary Bonsai 2 (1.76 bits per weight), Tencent’s Sherry (1.25-bit ternary, arXiv 2601.07892, shown on models up to 1B), and dlab’s runtime dynamic compression (1 to 2 bits, perplexity only). These are the authors’ own results; we know of no independent check.
The travel diary
The weights are the map. The KV cache is the diary: for every token already read or written, the model stores a key and a value in each attention layer, and every new token looks back through the whole diary. Attention works like a flashlight: the current token’s query lights up the keys most related to it, and the values of those entries are blended into the answer. Keys do cluster by meaning, which is why the flashlight finds them.
The diary grows with every token. Its entries can be compressed: KIVI quantizes keys per channel and values per token, while TurboQuant rotates and quantizes them (arXiv 2402.02750, 2504.19874). Pages can be torn out, which is eviction: keep the recent tokens and the first few, and drop some of the rest. Entries can be smaller by design: multi-head latent attention, introduced in DeepSeek-V2, stores one compressed vector per token instead of full keys and values. And paged and offloaded caches change how the diary is stored and where it is kept.
Models also learn to park some attention on the first tokens or on a learned “sink”, and removing it breaks them. gpt-oss has learned sinks. MLX’s stock quantized attention refused them, so quantizing its KV cache needed a kernel of its own.
Whether the map or the diary is the heavier load depends on who the machine is serving. Every decode step reads the weights (one copy, shared by everyone the machine is serving) plus each conversation’s diary.
| Regime | What dominates | What helps most |
|---|---|---|
| One user, short context (a laptop) | the weights: every token rereads the active model | weight quantization, almost proportionally, when the kernel is memory-bound; mixture-of-experts, because only active experts are read |
| One user, long context (100K tokens, or a long reasoning trace) | the diary catches up with and passes the map | KV quantization, sparse attention, short-diary architectures, eviction |
| Many users (server batching) | weights are read once for the whole batch; diaries grow with users x context | KV compression and paging decide how many users fit |
| Reasoning models | the answer itself is long, so both the diary and the token count grow | measure time to answer, not tokens per second |
To put a number on the diary: a Llama-3.1-8B-shaped model (32 layers, 8 key-value heads, head size 128, 16-bit values) stores 32 x 8 x 128 x 2 x 2 bytes, about 131 KB, per token. At 128K tokens that is about 17.2 GB (131,072 tokens), roughly the size of the model’s own 16-bit weights. gpt-oss-20b is lighter because half its layers only look back 128 tokens: 24.6 KB per token for the 12 full-attention layers, 0.8 GB at 32K.
What the picture hides
Faster steps can mean a longer walk: aggressive quantization can make reasoning models think longer, which can cancel the per-token speed-up (arXiv 2606.25519), so the number to measure is time to a correct answer. And a part made much faster in isolation can change the whole almost nothing if it was a small share of the time.
Correctness needs its own check. Some maps are simply broken: published quantized models can be silently defective (5 of 305 official Ollama models in one audit, arXiv 2609.05881). Even matching scores can hide damage. Two models can score the same on a benchmark while disagreeing on many answers; divergence measures (KL) see it, and small test sets cannot.
The metaphor has limits of its own. Weights are not concepts: the landscape of numbers and the landscape of meaning are linked only through the whole computation. High-dimensional space is not three-dimensional, and some of its behaviour (almost every direction is orthogonal to almost every other) has no everyday analogue. And every two-dimensional picture of the meaning landscape is a shadow that distorts distances.
The frame still serves the question the rest of this article is about: how far can a model be compressed for the work it has to do, and which errors are we willing to accept, including on rare requests?
Part 2: one laptop, one model, five questions
The model is gpt-oss-20b, OpenAI’s open-weight reasoning model, on a MacBook with an M4 Pro and 48 GB of memory, run through Apple’s MLX framework. It stores most of its knowledge in 32 “experts” per layer, of which each token consults four, and OpenAI ships those experts already compressed to 4 bits. That makes it a useful place to ask practical questions, because the builder has already done the easy compression.
The work went through five questions, in this order. Can land the builder already terraced be terraced further? Does each token need all the land it walks over? If not, can the hardware skip the rest? How would we know if any of it made the model worse? And can the model stop reading once more reading cannot change the answer? The fourth question limits what we can claim about the others. The fifth holds the results we kept.
Where the bytes go
Before compressing anything, estimate what each generated token reads from memory. At one user and short context, the weight bytes a token touches add up to about 3.71 GB (an estimate from the model’s layout, not measured traffic). About a third of that is the four active experts, a third is attention, and a third is the output head: the final matrix that turns the model’s internal state into a score for each of its 201,088 vocabulary entries. The experts take 10.2 of the 13.8 GB on disk but only a third of what each token reads.
Show the numbers
| Where | Experts | Attention | Output head | Total |
|---|---|---|---|---|
| On disk | 10.2 GB (74%) | - | - | 13.8 GB |
| Read per token | 34% | 34% | 31% | 3.71 GB |
The first step is well known: store attention and the output head at 8 bits instead of 16. Decode speed rose from 53.9 to 69.4 tokens per second, +29%, while prefill barely changed. The estimated bytes fell by a factor of 1.44, so byte arithmetic predicted more; the 8-bit path reached a lower effective bandwidth than the 16-bit one. We did not profile why. A community build already uses the same idea, with 8-bit embeddings as well; we add paired quality evidence, in Question 4. Later comparisons call this version the 8-bit incumbent.
Question 1: re-terracing land the builder already terraced
The experts ship in MXFP4: each weight is one of 16 values, with a shared power-of-two scale for every block of 32. Could a better quantizer squeeze them to 3 bits?
The builder used the terraces well. The 4-bit codes carry 3.86 of 4 possible bits of information, and one of the 16 codes is effectively a duplicate (+0 and -0). There is almost no slack left.
We then rebuilt every expert through TurboQuant at 3 bits: rotate each group of 128 weights with a Hadamard transform, then quantize with levels placed for a bell curve. In a separate numerical probe the rotation alone was harmless: undoing it restores the weights to within 1.2e-7. The quantization is not. The rebuilt weights differ from the vendor’s by 18% in every layer and every projection, flat across depth. That is almost exactly what 3-bit Lloyd-Max quantization gives on a Gaussian (18.6%): after rotation, the quantizer sees about the error it would see on generic bell-curve data. That does not prove no exploitable structure survives, but this quantizer did not find any.
And yet, on 200 MATH-500 competition problems, the 3-bit rebuild solved 186 and the original solved 188. That is inside the noise we measured later (see question 4), so read it as “no measurable loss at this resolution”, not “free”. An 18% weight error that a 200-problem test cannot see is a statement about the test as much as about the model.
Three limits on this result. We stored the rebuilt weights in an 8-bit container so the model could run them, so there is no speed or memory claim. The comparison measures reconstruction plus container; without a transform-and-container control it cannot isolate TurboQuant’s own contribution. And in a layer-level probe, TurboQuant at 4.34 bits, more than the original 4.25, still moved the experts’ outputs by 6-10%: for the requantizers and budgets we tried, changing the terrace spacing damaged land that was already terraced.
Question 2: the wind
A weight matrix is multiplied by the token’s activation vector, one number per input column. Most of those numbers are small. If a column’s activation is near zero, its weights contribute almost nothing to this token’s output, whatever precision they are stored in.
Call the activation the wind. It changes with every token, and it is gusty in the extreme: a handful of channels blow thousands of times harder than the rest. Our budget search only worked once it searched on a log scale.
So instead of storing each column at one fixed precision, let each token read each column in one of three states:
- skip: don’t read it;
- coarse: read only the high half of each stored number;
- full: read it all.
A column’s state depends on how hard the wind blows on it times how much its weights matter. For the experts, the coarse state can reuse the builder’s own grid: drop the lowest bit of each 4-bit code and read the retained code as the midpoint of the pair it stands for, so no new tables are needed.
In a provisional layer-level probe, three states beat skipping alone and both beat re-quantizing everything to 3 bits at equal bytes read. This is related to known work: TEAL skips by activation magnitude, WiSparse scores columns by activation times weight norm, and 2511.04477 combines dynamic sparsity with quantized weights. What we tested is the three-state version, with the builder’s grid as the coarse tier.
To compare versions, we feed the same text through each and measure how far apart their next-word probabilities are. This divergence is sensitive and cheap, but it does not say whether answers get worse. On gpt-oss-20b’s development traces, measured against the 8-bit incumbent, the cheapest land to thin was the output head: its reads dropped from 8.5 to 4.5 bits per weight at a divergence of 1.7e-3. That result is also provisional. (8-bit attention’s own divergence, also 1.7e-3, was measured against a different reference on other traces, so the two cannot be ranked.)
Question 3: can the hardware skip it?
The plan saves time only if the GPU skips those bytes. With MLX’s stock kernels, on the shapes we tested, a sparse read only beat a dense one when the skipped region was a tile of hundreds of adjacent columns. Quality, though, needed per-column decisions in our tests: for the experts, grouping even two adjacent columns roughly tripled the error, and in the combined recipe 576-column tiles raised divergence about 40-fold.
The way out was to store each matrix column by column, so each column is one contiguous chunk of memory, and write Metal kernels that read only the selected columns. In isolated benchmarks with chosen column mixes, a first version reached 0.95x of the tuned dense kernel at full read and 1.64x at half the columns, and the three-state kernel ran the output head 1.51x faster at 5.5 bits read and 2.27x at 3.5. These exclude the production selector’s full cost. One large matrix turned per-column savings into time almost in proportion.
The experts did not follow. The best expert kernel broke even at 69% of columns in isolation and lost in the running model (65.4 vs 68 tokens per second). Per-call selection and launch overhead across many small expert calls is the plausible cause; we did not isolate it. Attention, also small matrices, gained nothing end to end.
So the one surface where skipping paid was the output head: +9.9% decode speed at 4K context (69.4 to 76.3 tokens per second, medians of run medians) and +6.4% at 32K, reading 3.9 bits per head weight instead of 8.5. The second of two independent reviews found three defects (a kernel reading past the end of an array in padded lanes, two different calibrations for quality and timing, and a “null” setting that was not null), all fixed before these numbers. Metal’s shader validation found the out-of-bounds read; the numerical tests had passed.
The tie
Then the control: a plain 4-bit output head. Ordinary quantization, no wind, no custom kernel.
It is about as fast: 75.8 tokens per second against 76.3.
The adaptive head’s advantage is fidelity, not speed. Measured against the 8-bit reference, it diverges half as much as the plain 4-bit head (KL 3.1e-3 vs 5.9e-3), while reading slightly fewer head bytes (3.9 against 4.5 bits per weight), placed where the wind blows. It costs about 0.9 GB more peak memory than the plain 4-bit head on gpt-oss (0.42 GB on MiniCPM), because the prefill path still uses the ordinary 8-bit head.
On a second model the gap widens. MiniCPM5-2B (OpenBMB, released in September 2026) is a small dense model. On its own development traces a plain 4-bit head diverges 2.6 times as much as gpt-oss’s does on its own, and the adaptive head keeps its divergence to 43% of the plain one. Two models on different traces do not make a general law about small models. Speed again follows the head’s share of the bytes each token reads: 12% on MiniCPM against 24% on gpt-oss, and a correspondingly smaller gain (+5.3% vs +9.9%). Time saved was about 79% (MiniCPM) and 70% (gpt-oss) of the estimated bytes saved: accounting ratios over modeled weight bytes, not measured memory traffic.
Question 4 asks whether half the divergence makes a measurable difference on a task.
Question 4: how would we know?
Divergence cannot say whether answers get worse. For that we ran MATH-500: 200 problems, one sampled answer per problem, graded by a rule-based checker, with the same random seed per problem in every arm so the comparison is paired.
The results on gpt-oss: native 188, 8-bit incumbent 186, adaptive head 186, plain 4-bit head 184, 3-bit experts 186. Paired score intervals on the differences ran from about -5.8 to +3.8 points. Every comparison stayed INCONCLUSIVE against the declared 2-point margin, except the plain 4-bit head against native, which the frozen point rule marks FAIL at exactly -2.0.
Next we asked how much the test moves on its own. First we checked that the pipeline is repeatable (4 items, 1,024 tokens): the same prompt gives bitwise identical probabilities, and the same seed gives the identical answer, in one process, across processes, and with another job loading the GPU. So re-running an arm with the same seeds reproduces it exactly, and tells us nothing.
Any change to the model, even one that moves a probability in the fourth decimal, changes one sampled word somewhere in a thousand-word solution, and from there the two answers take different paths. They are not independent (same question, mostly the same model), but they are no longer the same answer. The pairing in a paired test comes from the questions, not from the seeds. To see how much the test moves on its own, run the same arm with different seeds.
So we ran the incumbent three times with three seed sets: 186, 188, 183. Between seed sets, 8 to 13 of the 200 problems flip between right and wrong. Between arms, 10 to 12 flip. Across the three seeds, 177 problems were always solved, 6 never, and 17 depended on the sample: about one problem in twelve is decided by sampling. The arm differences we measured (0 to 2 points) are the same size as the incumbent’s own seed-to-seed spread (183 to 188). That range describes the noise; landing inside it does not show an arm is equivalent.
Show the numbers
| Version | Solved of 200 |
|---|---|
| As shipped | 188 |
| 8-bit head and attention | 186 |
| Adaptive head | 186 |
| Plain 4-bit head | 184 |
| 3-bit experts | 186 |
| 8-bit head and attention, three seed sets | 186, 188, 183 |
How large a difference this design can reliably detect depends on assumptions; with the observed 5-6% discordance, a paired normal approximation at 80% power gives about 4.4-4.9 points. The test leaves the adaptive head’s effect on accuracy unresolved; it does not show the effect is small. OpenAI’s own evaluation code samples each MATH problem 10 times by default.
On MiniCPM a different confound showed up first. The small model thinks at length: about a fifth of its answers ran into the 8,192-token limit. The adaptive head got 45 of 200 wrong, and 41 of those hit the limit. Whether they would have been right with more room is unknown; at that limit the test largely measures whether the model finishes within budget. Re-running only the cut-off problems at 16,384 tokens, same seeds, added 18 to 23 correct answers per arm (A 158 to 181). With the larger budget every paired interval still spans zero: the adaptive and plain 4-bit heads end at 176 and 175 of 200. The budget moved accuracy more than any head did. That re-run relies on the runs repeating their first 8,192 tokens exactly, which we observed but did not prove.
Question 5: can the model stop reading once more reading cannot change the answer?
The adaptive head chose what to read from statistics. A reviewer suggested a sharper question: start with a cheap coarse pass, then read more only where it could still change the result, with a rule that says when to stop.
The output head, exactly. Read only the high 4 bits of each stored 8-bit weight. That gives every vocabulary row a coarse score and a bound on how far the missing low bits could move it. Rows whose best case cannot reach the leader’s worst case are dropped; the rest are read in full. If one row’s worst case beats every other row’s best case, the chosen token is the one the full head would choose. The few steps where intervals overlap are resolved exactly on the CPU, two steps behind the generation, so the fast path never waits.
Getting the bound right took five tries: a reviewer broke each of the first four. One used the wrong rounding unit for 16-bit floats, so a true winner was pruned and a wrong token was “certified”. Each failure is now a regression test.
This version emits, for every greedy or sampled token, exactly what the full 8-bit head would have chosen. On 396 checked positions all 396 agreed with a high-precision recomputation, and in 12,288 generated tokens the delayed check never had to correct one. Our first timing said 7.3% faster than MLX’s stock path. Most of that was the first prompt paying for warm-up. Timed fairly, it is 2.2% faster. On a second, smaller model it was exact again and 6% slower. So exactness is free on this model, but it is not a speed-up.
The diary, approximately. For the KV cache the same idea changes shape. Every score needs every key, and bounding keys pruned almost nothing. What pays is reading values only where attention lands: a fused kernel scores all keys, then reads each token’s value in full, coarsely, or not at all. This is an approximation, not a certificate.
Per token it decodes 1.23 times as fast at 64K tokens of context. Per request it is 1.3% faster.
Show the numbers
| Arm | Reading the prompt | Writing the answer | Total |
|---|---|---|---|
| Exact attention | 130.9 s | 29.6 s | 160.5 s |
| With the cache | 132.3 s | 26.1 s | 158.4 s |
At 64K, reading the prompt takes about 131 of each request’s 160 seconds, and the cache does not touch that. Decoding is 18.5% of the request, so even a 1.23 times faster decode can gain at most 3.5%. Sampled answers also differ in length from run to run, and that moved the request time more than the faster step did: one night the cache’s answers were 4% shorter and it measured 5.5% faster, the next they were 8% longer and it measured 1.3%.
Answer quality. Under greedy decoding the answers looked worse: on two of eight long answers the cache never stopped. Those turned out to be repetition loops, and exact attention falls into them too.
One detail had already cost us a comparison. gpt-oss’s chat template writes today’s date into every prompt, so the same question is a different prompt on a different day, and greedy answers change with it. We found it when the unmodified model disagreed with itself across two nights. The date is now pinned.
gpt-oss is meant to be sampled, so the next test samples, with the same random seed for both arms of each question.
Show the numbers
| Run | Exact attention | With the cache |
|---|---|---|
| Night 1, 32K context | 29 of 40 | 28 of 40 |
| Night 2, 64K context | 28 of 40 | 26 of 40 |
| Night 3, 32K, new seeds | 25 of 40 | 31 of 40 |
| Night 3, 64K, new seeds | 28 of 40 | 29 of 40 |
| All four | 110 of 160 | 114 of 160 |
Two nights leaned against the cache by a few answers. A third night, with fresh seeds, leaned the other way. Had we stopped after two nights we would have written “slightly worse”; after only the third, “clearly fine”. Neither was in the data. Pooled, the cache has 114 of 160 right and exact attention 110.
The more useful number is how often an answer flips between right and wrong.
Show the numbers
| Comparison | Flip rate | Flips |
|---|---|---|
| Cache vs exact attention, same seed | 22.5% | 36 of 160 |
| Exact attention vs itself, new seed | 25.0% | 20 of 80 |
| Cache vs itself, new seed | 27.5% | 22 of 80 |
The cache and exact attention disagree on 22.5% of questions. Exact attention disagrees with itself on 25% when only the random seed changes. The cache changes answers no more often than rolling the dice again does, and only one of the 80 questions flipped the same way on both nights. The 160 answers show no sign of a difference and rule out a loss of more than 5 points, but not a loss of 2. That does not pass our rule.
What this adds up to
Re-terracing the builder’s 4-bit grid to 3 bits moved the weights by 18%, and a 200-problem MATH-500 test could not resolve the accuracy difference. Because the rebuild ran in an 8-bit container, it makes no speed or memory claim either. Skipping columns gained nothing end to end on the experts or on attention: where skipping pays is decided by matrix size, not only by how sparse the wind is. For the many small expert calls, selection and launch overhead is the plausible cause; we did not isolate it.
The output head was the one place skipping paid: about +10% decode speed on gpt-oss-20b. Ordinary 4-bit quantization of the head was about as fast. The adaptive head’s advantage is fidelity: its divergence was 43-53% of the plain head’s, at 0.4-0.9 GB more peak memory. No task we ran resolves whether that changes answers. The evaluation limited the accuracy claim; the implementation’s speed and memory costs limited its practical value.
Asking whether more reading could change the answer paid off more. On the output head it gave tokens, greedy or sampled, that are exactly the full 8-bit head’s choice, at no net cost on gpt-oss (about 2% faster than stock on reasoning text); on a small fast model it was 6% slower. On the diary it gave value reads that follow attention at long context. Those are approximate: 1.23 times as fast per token at 64K but only about 1% faster per request, because reading the prompt dominates whole requests at long context on this machine. Whether that reading is limited by compute or by memory has not been profiled. Over 160 sampled answers the cache showed no sign of an accuracy difference (114 against 110 right; a loss above 5 points ruled out). On long greedy answers it changed which answers fell into repetition loops, as any small change does; exact attention loops too.
We declared 2 points as the loss we would accept. Neither test could resolve a difference that small: not the 160 sampled answers, and not the 200 problems with one sample each. The answer to “is it worse?” stayed “we can’t tell”.
Part 3: who improves the improver
An automated research loop has to tell improvements from noise, and most of this campaign ran as one. A person set the objective, the quality gates and the budget; an AI agent proposed changes, wrote the kernels, ran the measurements overnight and recorded the results; an independent AI reviewer checked each stage.
OpenAI and Anthropic have measured AI use in their own research. OpenAI reported in September 2026 that, as of mid-August, its research organisation as a whole used 3.1 agent-workdays of effort for every workday of human labour (total agent runtime, not per researcher).
Anthropic reported in September 2026 that, as of August, Claude “leads” 26% of its AI R&D work measured on a weighted task basket, that over 90% of the work involves at least substantial AI collaboration, and that no measured subset runs fully autonomously. Its ratings come from a Claude agent and a Claude judge, and Anthropic itself flags that the judge could share the model’s errors. (Sources: OpenAI “Research acceleration” 2026-09-06; Anthropic Institute “Measurements for understanding the pace of AI development”.)
Some research now closes the loop, with the agent improving the harness it runs in. AIDE² (arXiv:2609.26457) lets an agent rewrite its own harness, evaluates each rewrite on hidden tests, keeps improvements, and lets the improved agent take the next turn; one eight-day run accepted 7 rewrites and moved its grade from 0.703 to 0.778. SIFT (arXiv:2609.19526) makes this cheap by letting an LLM judge rank candidate patches and running the expensive benchmark only on the best-ranked ones.
Both face the problem we did: telling an improvement from noise. AIDE² accepts a rewrite if its averaged hidden score beats the incumbent’s, with no stated margin; its authors write that “a single noisy comparison can derail the outer loop’s subsequent search”, and they test transfer to unseen benchmarks separately, which is the right check. SIFT reports that four repeated searches with one backbone ended between 32.0% and 35.6%, a spread of whole searches, while its margin over the previous method, measured with a different backbone, is 0.6 to 1.5 points with no repeat spread reported. Neither shows that its accepted steps were noise, and neither states how its acceptance rule would detect that.
The evaluator is also within reach. SIFT’s authors observed candidate patches that “relaxed the evaluation harness itself (e.g. extending timeouts or retry counts)” and blocked them; none reached their final experiments. AIDE²’s final agent patched a held-out scoring script that crashed whenever one test failed: a needed repair, but still a change to the evaluator made by the thing being evaluated.
Our own evaluator also missed defects. Each of seven review rounds over the campaign found at least one defect. In the last three alone: a bound that could certify a wrong token, a crash on the second block append, and a speed-up that was mostly the first prompt paying for warm-up. The tests we wrote passed before every one of those reviews. The machine itself moved under the measurements: the GPU is shared with whatever else runs in the daytime, and the operating system was upgraded between two timing runs. None of this is visible in a single score.
Generating candidates is cheap. The scarce part is an evaluator that knows when it cannot tell two things apart:
- a null arm that uses fresh randomness, not repeated seeds;
- the smallest difference worth detecting, declared before the run, with the sample size to detect it;
- enough samples per item that per-item luck averages out;
- an acceptance margin tied to measured noise;
- and an evaluator outside the reach of the thing being evaluated.
These conditions are necessary but not sufficient. They make an unattended loop’s decisions auditable, not safe.
Compression tricks age fast: Google now ships Gemma 4 checkpoints trained for 4-bit deployment, which takes away some of the room post-hoc quantization had, though not the case for reading fewer bytes at run time. An evaluator with those properties ages more slowly than any compression method it judges.
Appendix: the full trail of Question 5
This working account gives every reversal in Question 5 in the order it happened.
The long version of Question 5: can the model stop reading once more reading cannot change the answer?
The adaptive head chose what to read from statistics. A second reviewer suggested a sharper question: start with a cheap coarse pass, then read more only where it could still change the result, with a rule that says when to stop.
For the output head the rule can be stated exactly, and getting it right took three tries. Read only the high 4 bits of each stored 8-bit weight. That gives every vocabulary row a coarse score and a bound on how far the missing low bits could move it; the bound must also cover floating-point rounding in both passes. Rows whose best case cannot reach the leader’s worst case are dropped; the rest are read in full. If one row’s worst case beats every other row’s best case, the chosen token is the one the full 8-bit head would choose in exact arithmetic; when intervals overlap (near-ties, exact ties) the step is flagged and nothing is claimed.
A reviewer broke our first four versions. The first ignored rounding in our own refinement, so an exact tie was “certified” arbitrarily. The second used the wrong unit roundoff for bf16 (2^-9 instead of 2^-8): a bias of 1 + 2^-8 rounds to 1.0, the true winner was pruned, and a wrong token was certified. The third extended the rule to sampling (Gumbel-max: add random noise to each score and take the largest) and forgot that adding noise of order one rounds away a gap of 2^-42 between two rows. The fourth certified 97% of steps and returned an unchecked choice for the rest without saying so. Each case is now a regression test.
This version resolves every flagged step exactly: integer arithmetic on the CPU over the few rows still in contention, with the same noise. Every emitted token then equals a declared reference, the 8-bit head’s logits in exact arithmetic plus the drawn float32 noise, lowest index on ties. On 396 teacher-forced positions all 396 agreed with a float64 recomputation, 9 of them only after resolution. The simple way to wait for the check, a host sync per step, made decoding slower than stock (2-10% in two uncontrolled runs), because it breaks the pipelining that hides the CPU’s work. So the check now runs two steps behind: emit the fast choice, read the certificate once it has been computed anyway, and regenerate from that token if exact resolution disagrees. That never happened in 12,288 emitted tokens; 366 steps needed resolution and all confirmed the fast choice.
Under a controlled protocol (warm-up per arm, both arm orders per prompt, the same noise for every sampling arm, run at night on an otherwise idle laptop), exact decoding was 2.2% faster than MLX’s stock sampler for greedy decoding (per prompt -4.0% to +1.5%) and 2.8% faster than a dense head drawing the same noise at temperature 1, on 512-token outputs with the end-of-answer token ignored. Leaving flagged steps unresolved would be 4-5% faster but not exact. Our first timing reported -7.3%; most of it was the first prompt paying the warm-up cost, and a reviewer caught it. So exactness is roughly free here, not a speed-up. On a second model, MiniCPM5-2B, it was exact again (385 of 385 checks) but 6% slower. It takes about 20% less time per token than gpt-oss (11.1 against 13.9 ms, prefill included), and its head is a smaller share of each token’s bytes; why the saving turns into a loss there is not yet measured. The cost depends on the text (one prompt per kind of text): on repetitive filler about half of the rows stay in contention and the head is slower than dense; on MATH answers about 2% do. These are six MATH prompts and a handful of other prompts on one model and one machine.
The same idea on the KV cache (the diary) changes shape. Every score needs every key, and bounding keys pruned almost nothing. What does pay is reading values only where the attention weight lands. A fused Metal kernel scores all keys (stored at 8 bits per channel), then reads each token’s value in full, coarsely, or not at all, depending on a conservative estimate of its weight. This is an approximation, not a certificate: the estimate never skips a value whose weight under the quantized keys reaches the threshold, but many small skipped contributions can add up, and nothing here guarantees unchanged attention output or tokens.
Measured on 40 long-context dev items at 16K and 32K: with 64-token key-scale pages the kernel gave the same final answer as exact attention on all 40 (31 correct each); with the 128-token pages we then adopted, 39 of 40 matched and it scored 32 against 31. Plain 8-bit KV reading every value scored 30. In isolation, the page-128 kernel ran about 1.5 times as fast as MLX’s bf16 attention on a captured 64K query and cache. In the running model, after two fixes a reviewer and a timing run forced (a crash on the second block append, then a copy of the whole cache every step), the page-128 cache made 64K decoding 1.24 times as fast (steady step 27.6 to 22.0 ms) and 32K decoding 1.08 times as fast. Those numbers come from 600-token outputs with the end-of-answer token ignored; the needle answers themselves end after about 40 tokens, where the whole request is prefill-bound: whole requests 0.99-1.00x. On answers that end naturally (high reasoning effort, up to 4,096 tokens, 8 items at 32K and 64K) the cache was again 7-21% faster per token, and on two items it never stopped: it ran to the 4,096-token cap where exact attention answered in about 720 tokens. That looked like a quality problem, so we split it apart the next night. Every answer that hit the cap was a repetition loop, and exact attention loops too (2 of the 8, and one item loops in every variant). Reading every value, 8-bit keys alone, 8-bit values alone, both together: one variant looped on exactly the items exact attention did, the other four on larger sets, up to 5 of the 8, and none answered fewer correctly than exact attention by more than one item. Greedy decoding over a long answer is chaotic: any small change in attention picks a different path, and some paths loop. Eight items cannot say whether the cache makes loops more likely, and they cannot attribute the loops to anything; the pooled counts hint at more loops weakly (21 loops in 48 perturbed answers against 5 in 16), with no difference in correct answers. They do show that faster per token does not necessarily mean faster per answer when answer lengths change.
gpt-oss is meant to be sampled, not run greedily, so the next test samples at temperature 1, with the same random seed for both arms of each item. On 40 items at 32K the cache got 28 right against exact attention’s 29 (4 and 3 items right in only one arm), ran to the cap on 5 against 4, never on the same item, and wrote answers of the same average length (1,588 against 1,570 tokens). None of the capped answers was a loop: sampled, the model just kept reasoning. The requests took 2.5% less time in total, but a reviewer pointed out that exact attention had run first on every item, so that was not a fair comparison.
The next night repeated it at 64K, where the cache is fastest: 26 right against 28. Two nights, both leaning against the cache by a few answers. Then a third night reran both lengths with fresh random seeds and the arm order alternated: 31 against 25 at 32K, 29 against 28 at 64K. Pooled over all 160 paired answers the cache has 114 right and exact attention 110, an interval of -5 to +10 points. Had we stopped after two nights we would have written “slightly worse”; after only the third, “clearly fine”. Neither was in the data. What 160 answers support is narrower: no sign of a difference, a loss of more than 5 points ruled out, a loss of 2 points not.
The fair timing is also smaller than it first looked. Per token the cache decodes 1.23 times as fast at 64K. Per request it is 1.3% faster at 64K and 5% at 32K, the same whichever arm ran first. Two things eat the rest. At 64K, reading the prompt takes about 131 of each request’s 160 seconds and the cache does not touch that, so even a 1.23 times faster decode can gain at most 3.5%. And sampled answers differ in length: that night the cache’s answers were 8% longer, the night before 4% shorter, which is why the same change had measured 5.5% then. We first blamed that 5.5% on noise in prompt-reading time; a fact check against the records showed the prompt-reading times agreed and the difference was all in how much each arm wrote.
A detail cost us one comparison: gpt-oss’s chat template writes today’s date into every prompt, so the same item is a different prompt on a different day, and greedy answers change with it. Every comparison above is within one night; the date is now pinned.
The cache also still keeps the bf16 copy beside the packed one, so it saves no memory
(scripts/mlx_kv_cache.py); mlx-lm’s own 8-bit KV cache already does that part.
Earlier, on 120 held-out items with short answers at low effort (the sealed final split, opened once for this frozen version), the cache scored 91, exact attention 91, and plain 8-bit KV 90; the paired interval is -4.4 to +4.4 points, too wide to call it equivalent at our 2-point margin. And the two shortcuts did not stack: on 32K and 64K needle answers the certified head made the cached model slightly slower, not faster.
Sources and records, 2026-10-18. Our own measurements come from the campaign’s findings log and its saved records. Those records are not yet public, so no measured number in this piece can be checked against them today. Model: gpt-oss-20b under MLX 0.31.1 on an Apple M4 Pro with 48 GB. External work is linked where the text cites it, and each link was opened on the publication date.
Edited later on 2026-10-18 for readability. Part 1’s ten sections became five and its lists became prose; no measurement changed. Three qualifications are now stated where the first version lost or blurred them: the output-head result in Question 2 is provisional, the seeds figure’s shipped model has MXFP4 experts, and the closing summary no longer suggests that the 3-bit rebuild’s effect on accuracy is known to be small.