How these numbers are measured
Every model card links to the exact raw result behind it. Files are commit-addressed under /benchmarks/<commit>/ and never silently overwritten; the shape is documented in schema-v1.json.
Memory (peak)
Strata loads models no-materialize (streaming) — it never assembles the full model in memory. The reported peak is the MLX peak allocation load-through-generation (loading the core + paging experts + generating), measured on the development host. It is a real measurement, not a working-set estimate, but it can still vary with mlx-lm version, context length, allocator behaviour, and OS — so we quote it with its configuration in each result file. Fits-resident models report their true resident footprint (no paging).
Three measurement categories
Every result declares which of three regimes it was measured under. We never blur them, and the flags below say exactly which one applies.
- 1 · Development-host benchmark
- No artificial memory limit. The model runs on this 128 GB host, so throughput is device-specific and there is no memory-pressure cliff. device_real:false, memory_constraint_enforced absent.
- 2 · Enforced-envelope benchmark
- Memory availability is physically restricted with a wired (mlock’d, compression-proof) ballast, but the host still has more RAM installed. Useful for validating memory fit and stability — not equivalent to a real device of that size: the GPU, bandwidth, SSD and thermals are still the host’s. measurement:"enforced-envelope", memory_constraint_enforced:true, but device_real:false and throughput_is_optimistic:true.
- 3 · Physical-device benchmark
- Actually run on a machine whose installed RAM matches the target. This is the only case where device_real:true and physical_constraint_validated:true.
Throughput (tok/s)
Device-dependent and, on the development host, optimistic — the target-RAM configs are benchmarked on a 128 GB machine with no memory-pressure cliff. An enforced-envelope run (category 2) does not change this: throughput stays optimistic because the silicon is unchanged. Physical constrained-device validation (category 3) is pending. Run strata bench on your own device for the real number.
Provenance flags
- device_real
- true only when the run happened on a machine whose installed RAM matches the target (category 3). false for host and enforced-envelope runs.
- memory_constraint_enforced
- true when a wired ballast physically restricted available memory for this run (category 2). It records what was genuinely proven — memory fit — and nothing about the device.
- physical_constraint_validated
- true only for validation on an actually-constrained physical device (category 3). An enforced envelope on a larger host keeps this false.
Fidelity (paged vs resident output)
Machine-independent: decode-mode, teacher-forced next-token comparison of the paged model vs the fully-resident model at budget = top-k (maximum paging pressure). Two honest tiers:
- Bit-exact
- Paged output is bit-identical to resident, deterministically, on any prompt (DeepSeek-V2-Lite, OLMoE, Mixtral, Phi-3.5-MoE).
- Near-exact
- Quality-neutral ~1-ULP graph-context drift — notbit-identical; 0–1 high-confidence flips (Qwen-family, GLM). Never called “100% fidelity.”
- Agreement is prompt-dependent
- A single agreement % is not a universal score — Qwen3-30B measures 95–98% across different prompts. The stable signals are top-1 agreement and high-confidence-flip count.
- Perplexity Δ is noisy on short refs
- On ~100-token references it can go slightly negative; that is short-sample variance, not paging “improving” the model.
Provenance
Each result records schema version, model ID + revision, architecture, quantization, streamed/resident mode, hardware + RAM, OS, MLX / mlx-lm versions, prompt set, generated-token count, target/enforced memory, measured peak, swap & compression deltas, SSD reads/token, throughput, and all fidelity metrics.
Source-available under BSL 1.1 (becoming open source on the Change Date). The residency-bias routing policy implements arXiv:2412.00099.