SPEED-Bench: unified benchmark for speculative decoding

SPEED-Bench arrives to put order in how we measure speculation in language models. How well does the idea of using a lightweight model to propose several tokens ahead — then letting the target model verify them in parallel — actually work? It depends a lot on the data, the service pattern, and the system. SPEED-Bench proposes a standard to measure it in a realistic way.

What is SPEED-Bench

Speculative decoding (SD) uses a draft model that’s lightweight to speculate multiple future tokens, which the target model then verifies in parallel. The neat part: you can improve throughput without changing the final model’s exact output distribution.

SPEED-Bench is a unified benchmark designed to evaluate SD under conditions close to production. It combines datasets with wide semantic variety, realistic context-length buckets, and a measurement framework that integrates production-grade inference engines like TensorRT-LLM, vLLM and SGLang.

Why previous benchmarks fail

• Small prompt sets and little semantic diversity.
• Short input sequences and batch size = 1, which don’t reflect real loads.
• Use of high-level inference stacks that hide system details.

In practice, draft quality and speedup depend on: semantic domain, text entropy, context length, batch size, and whether the load is memory-bound or compute-bound. SPEED-Bench is designed to capture those dependencies.

How SPEED-Bench is designed

SPEED-Bench combines three main elements:

• A splitting cualitativa (Qualitative) focused on semantic diversity to measure draft quality: acceptance rates (AR) and acceptance lengths (AL).
• A splitting de rendimiento (Throughput) to measure system-level speedups across different input sequence lengths (ISL) and high concurrency.
• A framework de medición that normalizes tokenization and formatting, and integrates with production engines to get comparable metrics.

Qualitative split

• They aggregated examples from 18 public sources and organized them into 11 categories: Coding, Math, Humanities, STEM, Writing, Summarization, Roleplay, RAG, Multilingual, Reasoning and QA.
• Each category has 80 samples, 880 prompts in total.
• To maximize semantic diversity they use openai/text-embedding-3-small embeddings and a selection algorithm that minimizes average cosine similarity between pairs within each category. That reduces redundancy and exposes domain-dependent behaviors.

Throughput split

• ISL buckets from 1k up to 32k tokens, to reflect long-context applications like code assistants and RAG.
• For each bucket: 1,536 prompts (512 per difficulty level: low, mixed, high entropy).
• Truncation and padding are controlled to keep prefill costs deterministic and avoid using random tokens.

Measurement framework

• Tokenization and formatting are done externally: inferences receive pre-tokenized sequences to remove silent differences between engines.
• It captures fine-grained measures from streaming responses: step latency, acceptance behavior, Output TPS and User TPS.
• Integration with TensorRT-LLM, vLLM and SGLang to compare under real conditions.

Practical example and tool output

Example run (Llama 3.3 70B Instruct as target, EAGLE3 as drafter, TensorRT-LLM, batch 32):

mpirun -n 1 --oversubscribe python3 run.py --model_dir meta-llama/Llama-3.3-70B-Instruct \
  --tokenizer meta-llama/Llama-3.3-70B-Instruct \
  --draft_model_dir yuhuili/EAGLE3-LLaMA3.3-Instruct-70B \
  --dataset speed --dataset_path data/speed/qualitative \
  --tp_size 8 --ep_size 1 --draft_length 3 --output_length 4096 \
  --engine TRTLLM --concurrency 32 --show_progress

Example output (summary):

• Acceptance Length Histogram: 1: 57385, 2: 36968, 3: 24441, 4: 61182
• Conditional acceptance rate: 1 -> 1.0, 2 -> 0.681, 3 -> 0.698, 4 -> 0.715
• Acceptance Rate promedio por categoría (ejemplo): overall average AR = 2.4511
• Output TPS: 2518.15 (Output TPS/gpu = 314.77)
• E2E Request Time (mean): 4.7313 s
• TTFT Time (mean): 0.1217 s

They also compare AL and speedups by domain and model. Summary of the table shown in the paper:

These results make a few things clear:

• AL and speedups are highly domain-dependent: Coding and Math (low entropy) allow longer acceptances, while Roleplay and Writing (high entropy) are much harder to speculate.
• Lightweight methods like N-Gram can even cause slowdowns at moderate batch sizes.
• Native MTP (co-trained with the backbone) tends to achieve larger ALs than post-trained drafters like EAGLE3.

Important technical findings

SD evaluation must look at two things: draft quality by domain and system-level performance under real conditions.

• Vocabulary pruning in drafters (for example EAGLE3) reduces cost but can degrade AL on the long tail of inputs, especially in Multilingual, RAG and Summarization.
• Using random tokens to measure throughput is dangerous: it produces two main failures that distort results for SD and MoE:
  • Trivial Response: the model detects noise and replies with clarifications, inflating AL.
  • Topic Latching: the model latches onto keywords and generates coherent but misleading outputs, reducing AL.
• Quantitative example: measuring with random tokens can overestimate throughput by about 23% when SD is active.

Practical recommendations if you work with SD

• Evaluate both: acceptance length/acceptance rate and real throughput across different ISL and batch sizes.
• Avoid using random tokens as a proxy for prompt load.
• Use datasets with semantic diversity and avoid categories with too few samples.
• Prefer measurements that control tokenization and prompt formatting externally to compare inference engines.
• Keep in mind that vocabulary or head-layer optimizations can help in closed domains but harm generality.

SPEED-Bench is not just a dataset: it’s an ecosystem with datasets, a measurement framework and scripts ready to plug into SD pipelines. It serves both researchers who want to compare algorithms and production teams that need to understand real trade-offs between latency, throughput and output fidelity.

Original source

https://huggingface.co/blog/nvidia/speed-bench

Summary: SPEED-Bench is a technical, practical benchmark to evaluate speculative decoding (SD) under production-like conditions, combining semantic diversity and realistic throughput measurements. It includes two splits (Qualitative and Throughput) and a framework that normalizes tokenization and integrates with production inference engines.