Introducing LosslessBench

LZ Lily Zhang July 28, 2026

Every inference provider is racing to make open models faster. Nobody measures what the speedup costs. So I built LosslessBench, a benchmark across five domains, and ran GLM 5.2 and Kimi K3 through it. See Figure 1: both pages came from GLM 5.2, same prompt. The left was served by Z.AI at fp8, the right through Baseten's accelerated fp4 + spec endpoint. This is what inference acceleration can cost.

fp8 render of the photo card prompt: complete stylish photo card interface — Figure 1. Left: fp8 implements the photo card with the 3D flip. Right: fp4 renders a near-blank page.

fp4 render of the same prompt: near blank page with no photo card and no flip effect — Figure 1. Left: fp8 implements the photo card with the 3D flip. Right: fp4 renders a near-blank page.

The judge evaluates each rendered page on whether it fulfills the prompt: a photo card interface with a 3D flip on hover. fp8 built it, scored 93, and the judge wrote "excellently implements the 3D flip effect." fp4, in contrast, produced a page with no flip effect and no photo card at all, scored 15, and the judge wrote "fails to meet the prompt's requirements." fp4 model skipped the prompt requirement entirely.

Zoom out, Whoever handles inference acceleration owns the quality loss. GLM 5.2, accelerated by its vendors, loses 5.6 points on frontend design. Kimi K3, which owns its own acceleration, loses 0.3.

Figure 2. Frontend quality loss under acceleration: vendor-owned vs model-owned.

01Digging into Inference Quality Loss on GLM 5.2

Philip and Baseten have been incredibly transparent about their inference stack. When I asked what the Baseten GLM 5.2 endpoint actually runs, he answered in one line.

NVFP4 + spec dec + kv routing + pd disagg + more
— @philipkiely source

I built a benchmark called LosslessBench. It measures how much quality is lost between the model a lab releases, the endpoint an inference provider serves, and what the application layer receives.

It's the same GLM 5.2, just served differently:

Z.AI: fp8, the release precision
Baseten: NVFP4 + spec dec + kv routing + pd disagg

The domains are driven by OpenRouter token usage statistics across tasks. LosslessBench measures how inference loss impacts the majority of customers.

Figure 3. GLM 5.2 quality across five domains. Axes are independently scaled, so each domain's relative gap is visible.

Regarding Figure 3, each axis uses its domain's own benchmark and metric:

Frontend: OpenDesign, 100 prompts. Each generated page is rendered in a real browser, and a GPT-4o vision judge scores the screenshot on instruction alignment, aesthetics, and structure.
Creative: EQ-Bench longform score, judged over multi-chapter creative writing.
Guardrail: XSTest, classification accuracy on safe vs unsafe prompts built to sit near the decision boundary.
Coding: Terminal-Bench pass rate.
Agent workflow: tau3-bench long-horizon agent tasks, action match rate.

Notably, frontend design loses 5.6 points. We measured across 100 prompts, and the fp4 endpoint shows consistent, major degradation. The remaining four domains showed mixed results.

Eugene (@picocreator, CEO @ featherless.ai) mentioned that speculative decoding works very well on easy-to-predict tasks, but not on creative design and writing, because those tokens are hard to predict.

Here is the another failure. Prompt: "Stunning translucent calendar popup that smoothly blends into the interface." The fp4 page is visually appealing, but the calendar popup itself is missing. It scored 53. fp8 built the popup and scored 93.

Note that Baseten's published quality evidence for fp4 is BFCL, a function-calling benchmark. It tests functional correctness, not quality.

02Who Owns the Inference Quality?

Let's take a step back. Why has nobody measured inference quality before? Nobody has the incentive:

The model company wants distribution.
The inference provider wants speed, token volume, and token maxing.
The application layer buys whatever is fast, cheap, and functional.
The user sits at the very bottom and receives whatever comes out.

Figure 5. Every layer pushes its own incentive downward. The user gets squeezed at the bottom.

You can see token speed on every benchmark because it's exciting. Everybody wants to be token maxing. But you don't see inference quality metrics anywhere, because measuring them takes effort and integrity.

03Kimi K3 does QAT + draft model

I was mid running LosslessBench when Moonshot released Kimi K3, so I tested it.

Note that Kimi K3 is notably faster than Fable. I dug into why. Instead of releasing fp8 and letting vendors quantize, Kimi K3 does QAT from SFT onward and trains its own speculative draft model. It ships natively in MXFP4, benchmarks all reported at 4-bit.

Read the K3 tech report: K3 fine-tuned the pre-trained MTP layer into an EAGLE-3 draft model, to maximize acceptance rate under lossless speculative sampling.

This is a big shift. Kimi just moved that ownership back into the model layer.

When the model layer owns the inference acceleration, the thing left for a vendor is faithful execution. So I measured that: Moonshot's endpoint vs Baseten's endpoint. Both stacks have a speculative decoder, and about half of the tokens in every answer came from a draft model. 27 of the 61 output tokens in this call came from the draft model. You can see it in Baseten's API response:

"completion_tokens_details": {
  "accepted_prediction_tokens": 27,
  "reasoning_tokens": 4,
  "rejected_prediction_tokens": 0
}

Compare GLM 5.2: shipped fp8, vendors quantized to fp4 and trained spec drafts their own way. Everyone gets the speedup, but nobody owns the quality benchmark. Tax rolls downhill, customers eat it.

04Closing

K3 is not an outlier. It's a trend. DeepSeek pushed fp8 into pre-training. gpt-oss shipped MXFP4 with QAT. K2-Thinking reported every benchmark at INT4. K3 ships the speculator itself. Step by step, acceleration is moving from the serving layer into training.

Figure 6. The model layer absorbs acceleration step by step. The room left for serving shrinks toward one job: serve.

If this holds, models will leave the lab with nothing left to lose. Well, not exactly. A vendor can still relax the acceptance config: accept more drafted tokens, generate faster, sell more. We still need an all-rounded benchmark.

What's next for LosslessBench:

Full K3 evaluation across all five domains.
A controlled run: serve the model myself and toggle one acceleration at a time, so I can tell which one costs quality.
Measure token distribution shift: compare each token's probability against the reference.
Watch the acceptance rate: if a vendor relaxes the acceptance rule, acceptance goes up and quality drifts.
Make LosslessBench efficient enough to run as part of CI, catching regressions on every inference stack change.

A month ago I found GLM 5.2 running absurdly fast inside Devin Desktop and tweeted about it. Then the replies started asking the question I couldn't answer: does the speedup degrade the model? Two days later I presented a poster at the AIE conference: "Is speculative decoding all we need? Will GLM 5.2 dflash/dspark be lossy on long tail tasks?" This benchmark is the answer I owed those replies.

Full results across all domains → site