|
Paper brief · October 12, 2026 · 6 min read
|
|
Your LLM server was built for one model. Token routing needs two.
|
|
A NeurIPS 2026 paper from Tsinghua's thu-nics group explains why token-level routing has been stuck in notebooks, and ships a serving system that makes it 2–64× faster than what researchers had been running.
|
TokenRouter: Efficient Serving System for Token-Level LLM Routing
Fu, Liu, Wang, Dong, Ge, You & Wang · arXiv:2610.12242 · cs.CL
|
|
| |
|
If you read one paragraph
|
|
Letting a small model write most tokens and a big model write only the hard ones works remarkably well on paper. In practice it has been painfully slow, because vLLM and SGLang assume a single model owns a request. TokenRouter gives routing algorithms a three-function API, runs each model in its own subserver with asynchronous handoffs, and uses a Markov-chain throughput model to decide when each model should launch a batch. Result: the same routed tokens, decoded 2.01–64.15× faster.
|
|
| |
|
2–64×
decoding throughput vs. the stronger baseline, across 15 setups
|
|
~5%
of tokens sent to the 32B model for 32B-level quality (R2R)
|
|
3
functions a developer writes to plug in a new routing algorithm
|
|
| |
|
Why route per token at all?
|
|
The routing most of us know is query-level: look at the prompt, pick the cheap model or the expensive one, commit. ChatGPT's model picker, Cursor, and RouteLLM all work this way. It's simple, and it leaves a lot on the table, because a query is not uniformly hard. In a long chain-of-thought, most tokens are connective tissue and a handful are the ones that actually decide the answer.
|
|
Token-level routing lets a small model draft and a large model step in only on those tokens. The authors' motivating number: R2R matches a 32B model's quality while handing roughly 5% of tokens to the 32B, where query-level routing needs to send about 40% of whole queries. A whole family of algorithms works this way now: CITER, R2R, R-Stitch, and Co-LLM switch on an uncertain token; model ensembles (ME) pick a different model after every token based on fused distributions.
|
|
The catch: almost none of it has been served properly. The official R2R and CITER implementations don't even do continuous batching.
|
| |
|
Three ways your serving stack chokes on it
|
|
vLLM and SGLang are built around one assumption: a single model owns a request from its first token to its last. Routing at every step violates that in three distinct ways.
|
| 1 |
Step desynchronization. A 32B model takes far longer per decoding step than a 0.6B model. If you keep the two in lockstep, the small model spends most of its life waiting for the big one.
|
| 2 |
Batch admission delay. A routed request usually arrives while its target model is mid-step on another batch. It waits for that step to finish, then runs in a fragmented batch. Multiply by every handoff.
|
| 3 |
Implementation pain. There is no hook for per-step routing. Bolting one on collides with continuous batching and prefix caching, which is why most routing papers ship a research loop instead of a server.
|
|
| |
|
How TokenRouter fixes it
|
|
The design principle is request-centric programming, model-centric execution. You write the algorithm as if you were following one request around; the runtime does the ugly part.
|
|
On the developer side: three functions
|
| route() |
Runs after every decoding step. Returns a destination per request: 0 to keep decoding locally, otherwise the ID of a peer model. |
| send() |
Packages a delegated request: ID, the token suffix the peer hasn't seen, a position anchor, status, plus any algorithm-specific fields. |
| receive() |
Turns a returned peer request back into the local format. The default accepts all returned tokens; override only if your handoff semantics are unusual. |
|
|
R-Stitch on the small-model side is a few lines: compute normalized entropy in route(), hand off above a threshold, and have send() drop the uncertain token so the big model regenerates it. The large-model side is the same code with the comparison flipped. All five evaluated algorithms fit this shape.
|
|
On the runtime side: subservers, a third loop, and a pending state
|
|
Each model gets its own subserver, built on SGLang, with its own scheduler and private KV-cache pool. A single front door speaks the vLLM/SGLang client protocol, so it drops into existing deployments. Adding a third model means launching a third subserver.
|
|
A normal server has two loops: one talking to clients, one driving decoding steps. TokenRouter adds an inter-model loop that ships and receives peer requests. When a request routes away, it enters a new pending state: the local scheduler skips it, but its KV cache and serving state stay put. The local batch keeps decoding without it. When the peer's tokens come back, the request flips to running, appends the new tokens, and rejoins. No prefix re-matching, no KV reallocation. A model switch costs about as much as appending a token.
|
|
That alone kills problem 1. Problem 2 needs the clever bit.
|
|
Delayed batching, tuned by a Markov chain
|
|
An eager asynchronous scheduler launches a step the moment any request shows up, which means the next arrival waits a full step. TokenRouter's scheduler instead buffers incoming requests and launches only once B of them are waiting. Every subserver has its own B. Set it too low and you're back to admission delays; set it too high and requests sit in the big model's queue while the small model starves.
|
|
Rather than grid-search B, the authors model the whole routing process as a discrete-time Markov chain. Feed it the concurrency, the routing probability matrix between models, and each model's per-step latency, and it predicts throughput as a function of B. Pick the B that maximizes it. The one assumption: the number of tokens between consecutive handoffs is geometrically distributed, which they say holds for most of the algorithms tested.
|
| |
|
The numbers
|
|
Setup: one node of 8× A100-80G. Qwen3-0.6B paired with Qwen3-32B for CITER, R2R, R-Stitch and Co-LLM; ME adds Qwen3-8B as a third model. Three workloads: short AIME reasoning (~2k output), long AIME reasoning (8k output), and SWE-Smith agentic traces (~8k input, 1k output). Two baselines: each algorithm's official code, and a fairer “Std. Serving” setup the authors built themselves, with one SGLang server per model and an external dispatcher.
|
|
Official code vs. TokenRouter · each paper's own models & benchmark · concurrency 4
|
| Algorithm |
Official tok/s |
TokenRouter tok/s |
Latency |
| R2R (R1-Distill 1.5B/32B, AIME) |
89.6 |
244.6 |
751 s → 270 s |
| CITER (Qwen2 1.5B/72B, CSQA) |
17.2 |
149.3 |
7.6 s → 0.5 s |
| Co-LLM (LLaMA2 7B/70B, GSM8K) |
3.5 |
76.0 |
248 s → 11 s |
| R-Stitch (L1-1.5B / QwQ-32B, AIME) |
no code |
140.6 |
151 s |
|
|
Against the fairer Std. Serving baseline on the Qwen3 pairs, the range is 2.01–64.15× throughput and 2.03–63.64× lower end-to-end latency across 15 algorithm–workload combinations. The gap widens as outputs get longer: Std. Serving loses 58–85% of its throughput moving from short to long reasoning, while TokenRouter stays flat.
|
|
It also scales the way you'd want a server to. From 1 to 16 concurrent users, throughput grows 8.61× while each user keeps 51.7% of their single-user speed; official R2R manages 5.14× and 31.3%.
|
| |
|
Where the speedup actually comes from
|
|
The ablation is the most honest chart in the paper. R2R, Qwen3-0.6B/32B, concurrency 8, output tokens per second:
|
| Official R2R |
|
| TokenRouter, base |
|
| + extended CUDA graphs |
|
| + async dispatch |
|
| + delayed batching |
|
|
|
Read it carefully: the single biggest jump is CUDA graph coverage, which is engineering rather than a new idea. The two things the paper is actually about, asynchronous dispatch and delayed batching, add another ~60% on top of that. Both matter; neither is magic. The approach also holds up across pairs: 3.21× on 0.6B/8B, 2.56× on 0.6B/32B, 2.76× on 1.7B/8B, 1.99× on 4B/8B.
|
| |
|
What to keep in mind
|
|
This is a speed paper, not a quality paper. TokenRouter produces the same routed tokens the algorithm would have produced anyway, just faster. The quality claim is a Pareto plot (AMC23, Qwen3-0.6B/32B) showing token routing on TokenRouter beating RouteLLM-style query routing on SGLang; the text gives no accuracy numbers to check.
|
|
The headline 64× is partly a baseline story. Official implementations largely never had a server. The authors' own Std. Serving is the fair comparison, and it still loses by 2–64×, so the point stands, but anchor on the low end of the range for a well-engineered competitor.
|
|
Single node, single assumption. Everything runs on one 8-GPU box. The throughput model assumes geometric gaps between handoffs; algorithms that route in bursts are explicitly left for future work. And the big model's tensor-parallel size matters a lot while the small model's barely does, so budget GPUs accordingly.
|
| |
|
Why it matters
|
|
Token-level routing has looked like a research curiosity mostly because nobody could serve it. If 5% of tokens at 32B really buys 32B-level answers, and the serving overhead is gone, the cost math for long reasoning and agentic workloads changes. The interesting question now isn't whether to route per token, it's which router.
|
|
| |
|
|
| |
|
Running token routing in production, or trying to? Hit reply and tell me which algorithm and what broke. The best answers go in a follow-up issue.
|