Token Speed Maxxing Has a Price: LosslessBench on Inference Acceleration
Every inference vendor is racing to make open models faster. Nobody measures what the speedup costs. So I built LosslessBench: I benchmarked GLM 5.2 and Kimi K3 across five domains, measuring what inference acceleration would cost in quality, and why token speed maxxing has a price. See Figure 1: both pages came from GLM 5.2, same prompt. The left was served by Z.AI at fp8, the right through Baseten's accelerated fp4 endpoint. This is what inference acceleration can cost.
The judge evaluates each rendered page on whether it fulfills the prompt: a photo card interface with a 3D flip on hover. fp8 built it, scored 93, and the judge wrote "excellently implements the 3D flip effect." fp4, in contrast, produced a page with no flip effect and no photo card at all, scored 15, and the judge wrote "fails to meet the prompt's requirements." fp4 didn't implement badly; it skipped the prompt requirement.
Zoom out, and the whole benchmark comes down to one number per model. GLM 5.2, accelerated by its vendors, loses 5.6 points on frontend design. Kimi K3, which owns its own acceleration, loses 0.3. Whoever handles inference acceleration owns the quality loss.
01Digging into Inference Quality Loss on GLM 5.2
Philip and Baseten have been incredibly transparent about their inference stack. When I asked what the Baseten GLM 5.2 endpoint actually runs, he answered in one line.
NVFP4 + spec dec + kv routing + pd disagg + more
— @philipkiely source
I built a benchmark called LosslessBench. It measures how much quality is lost between the model a lab releases, the endpoint an inference provider serves, and what the application layer receives.
Same GLM 5.2, same prompts, same judge. Two endpoints:
- Z.AI: fp8, the release precision
- Baseten: NVFP4 + spec dec + kv routing + pd disagg
The domains are driven by OpenRouter token usage statistics across tasks. LosslessBench measures how inference loss impacts the majority of customers.
Regarding Figure 4, each axis uses its domain's own benchmark and metric:
- Frontend: OpenDesign, 100 prompts. Each generated page is rendered in a real browser, and a GPT-4o vision judge scores the screenshot on instruction alignment, aesthetics, and structure.
- Creative: EQ-Bench longform score, judged over multi-chapter creative writing.
- Guardrail: XSTest, classification accuracy on safe vs unsafe prompts built to sit near the decision boundary.
- Coding: Terminal-Bench pass rate.
- Agent workflow: tau3-bench long-horizon agent tasks, action match rate (small sample).
Notably, frontend design loses 5.6 points. We measured across 100 prompts, and the fp4 endpoint shows consistent, major degradation. The remaining four domains showed no major degradation.
Eugene (@picocreator, CEO @ featherless.ai) mentioned that speculative decoding works very well on easy-to-predict tasks, but not on creative design and writing, because those tokens are hard to predict.
The blank page is the failure you would catch. Here is the one you wouldn't. Prompt: "Stunning translucent calendar popup that smoothly blends into the interface." The fp4 page is visually appealing, the judge said so, but the calendar popup itself is missing. It scored 53. fp8 built the popup and scored 93.
Note that Baseten's published quality evidence for fp4 is BFCL, a function-calling benchmark. It tests functional correctness, not quality.
02Who Owns the Inference Quality?
Let's take a step back. Why has nobody measured inference quality before? Nobody has the incentive:
- The model company wants distribution.
- The inference provider wants speed, token volume, and token maxing.
- The application layer buys whatever is fast, cheap, and functional.
- The user sits at the very bottom and receives whatever comes out.
You can see token speed on every benchmark because it's exciting. Everybody wants to be token maxing. But you don't see inference quality metrics anywhere, because measuring them takes effort and integrity.
03Kimi K3 does QAT + draft model
I was mid running Lossless benchmark when Moonshot released Kimi K3, so I tested it.
Note that Kimi K3 is notably faster than Fable. I dug into why. Instead of releasing fp8 and letting vendors quantize, Kimi K3 does QAT from SFT onward and trains its own speculative draft model. It ships natively in MXFP4, benchmarks all reported at 4-bit.
Read the K3 tech report: K3 fine-tuned the pre-trained MTP layer into an EAGLE-3 draft model, to maximize acceptance rate under lossless speculative sampling.
This is a big shift. Kimi just moved that ownership back into the model layer.
When the model layer owns the inference acceleration, the thing left for a vendor is faithful execution. So I measured that: Moonshot's endpoint vs Baseten's endpoint. Both stacks have a speculative decoder; about half of the tokens in every answer came from a draft model. You can see it in Baseten's API response:
"completion_tokens_details": {
"accepted_prediction_tokens": 27,
"reasoning_tokens": 4,
"rejected_prediction_tokens": 0
}27 of the 61 output tokens in this call came from the draft model. With the speculator running, seven of ten prompts scored exactly identical. Mean gap: 0.3 points.
Compare GLM 5.2: shipped fp8, vendors quantized to fp4 and trained spec drafts their own way. Everyone gets the speedup, but nobody owns the quality benchmark. Tax rolls downhill, customers eat it.
04Closing
K3 is not an outlier. It's a trend. DeepSeek pushed fp8 into pre-training. gpt-oss shipped MXFP4 with QAT. K2-Thinking reported every benchmark at INT4. K3 ships the speculator itself. Step by step, acceleration is moving from the serving layer into training.
If this holds, models will leave the lab with nothing left to lose. Well, not exactly. A vendor can still relax the acceptance config: accept more drafted tokens, generate faster, sell more. There is always something in it for being unfaithful. We still need an all-rounded benchmark.
What's next for LosslessBench:
- Full K3 evaluation across all five domains.
- A controlled run: serve the model myself and toggle one acceleration at a time, so I can tell which one costs quality.
- Measure token distribution shift: compare each token's probability against the reference, no judge needed.
- Watch the acceptance rate: if a vendor relaxes the acceptance rule, acceptance goes up and quality drifts.
A month ago I found GLM 5.2 running absurdly fast inside Devin Desktop and tweeted about it. Then the replies started asking the question I couldn't answer: does the speedup degrade the model? Two days later I presented a poster at the AIE conference: "Is speculative decoding all we need? Will GLM 5.2 dflash/dspark be lossy on long tail tasks?" This benchmark is the answer I owed those replies.
