FOR SERVING & FLEETS
Serving optimization
Already running a Mixture-of-Experts model in production? These are real optimization runs: at a fixed memory budget, Strata races routing policies and deploys the one that moves the most tokens per second without breaking your quality budget. Every gain below is measured against standard routing — never estimated.
Just want to know what runs on your own machine? That's the laptop picker — a different question (does it fit & how fast) for a different user.
How to read a run
The headline is a throughput gain at a fixed memory budget — not an absolute speed. Same budget, better routing.
- Throughput gain
- Decode tokens/sec with Strata's chosen routing policy vs standard routing, at the SAME fixed memory budget. This is the IO-bound regime, where moving expert weights dominates.
- Policy
- The measured winner: cache (bias toward resident experts, arXiv:2412.00099), warm (recency), or hybrid. No policy is hard-coded — they race.
- Bias σ
- How hard the router is nudged toward already-resident experts, in router-logit standard deviations. Higher σ cuts weight movement but costs quality.
- Budget
- Expert weights held resident in fast memory. Everything else is paged. The whole race happens at a fixed budget.
- Quality cost
- Perplexity change vs the unbiased baseline. Every deployed policy stays under the run's stated quality budget — or Strata deploys nothing.
- Hard gate
- A GSM8K exact-match check (McNemar) run at the winning config before it deploys. A speed win that fails the task gate is rejected.
Optimization runs
ranked by measured gainGains are in the IO-bound regime (paging dominates). Absolute tok/s depends on hardware; the honest per-device number comes from strata bench on the target node. Residency-bias policy implements arXiv:2412.00099.