Documentation

Strata runs Mixture-of-Experts models larger than your RAM by keeping a budget of experts resident and streaming the rest from SSD. Install it once, then use it two ways: as a Python library inside your app, or as a CLI / OpenAI-compatible server.

Requirements: Apple Silicon Mac (M-series) · Python ≥ 3.10 · a sparse-MoE model in MLX format (GLM-4.5-Air, Qwen3-MoE, Mixtral, DeepSeek-V2, OLMoE, …). Dense or non-MLX models aren’t accelerated.

Installation

One install gives you both the strata Python library and the strata command-line tool. The [mlx] extra pulls in the Apple-Silicon runtime.

pip install 'amperor-strata[mlx]'

Contributors / from source: git clone https://github.com/amperor-org/amperor-strata then pip install -e 'packages/strata[mlx]'.

Use it in your app →

import strata and generate in-process. For putting a big local model inside a Python app.

Run & serve it →

The strata CLI to benchmark models and serve an OpenAI-compatible endpoint any app can call.

For apps

Use Strata as a Python library

After installing (above), import Strata and load a model bigger than your RAM — experts stream from SSD under the budget you set. This is the way to run a large local model inside an app on a machine that can’t hold it.

# pip install 'amperor-strata[mlx]'
import strata

# GLM-4.5-Air is ~60 GB on disk — load it under a 16 GB budget
model = strata.load("mlx-community/GLM-4.5-Air-4bit", target_ram_gb=16)

# one call, returns the full completion
print(model.generate("Explain how a CPU pipeline works.", max_tokens=256))

# or stream tokens as they arrive
for piece in model.stream("Write a haiku about caches"):
    print(piece, end="")

Measured on a 128 GB M5 Max: that ~60 GB model loads and generates at a ~11.5 GB peak under target_ram_gb=16 (peak scales with the budget you pick).

Good to know

• model accepts a Hugging Face repo id or a local MLX model directory.
• The first load of a model builds a paged expert-store once (cached under ~/.cache/strata); later loads stream from it.
• generate() / stream() also accept OpenAI-style messages and a temperature (0 = greedy).

Signature

strata.load(
    model,                 # HF repo id or local MLX dir
    target_ram_gb=None,    # fit the working set into N GB (default: this machine)
    budget=None,           # or set resident experts/layer directly
    store_dir=None,        # paged-store location (default: ~/.cache/strata/…)
    policy="none",         # routing/paging policy: none|warm|cache|hybrid
) -> Strata

Strata.generate(prompt, *, max_tokens=256, temperature=0.0) -> str
Strata.stream(prompt,   *, max_tokens=256, temperature=0.0) -> Iterator[str]

For the terminal

Run & serve from the CLI

The same install gives you the strata command. Use it to benchmark a model on your machine, or to serve it behind an OpenAI-compatible endpoint.

Benchmark a model

Downloads the model, builds its expert store once, and reports fit + throughput + fidelity. --target-ram previews a smaller device.

strata bench --model mlx-community/Qwen3-30B-A3B-4bit
strata bench --model mlx-community/Qwen3-30B-A3B-4bit --target-ram 16

Serve an OpenAI-compatible endpoint

Binds 127.0.0.1:8399 and exposes the API under /v1. The config is a flat key: value file (produced by strata optimize): model, store, expert_budget are required; policy / prefetch optional.

strata serve --config strata-deployment.yaml --host 127.0.0.1 --port 8399

Call it from any OpenAI SDK

Point base_url at the server — no code changes, no custom SDK. No API key is required (pass any placeholder).

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8399/v1", api_key="strata")
resp = client.chat.completions.create(
    model="strata",  # the server serves its configured model
    messages=[{"role": "user", "content": "Explain a CPU pipeline."}],
    max_tokens=256,
)
print(resp.choices[0].message.content)

Reference

HTTP API & CLI reference

HTTP endpoints

POST /v1/chat/completions: OpenAI-shaped. Reads messages (required), max_tokens (512), temperature (0.7; ≤0 = greedy), stream. Non-stream returns a chat.completion with an extra usage.forge_decode_tok_per_s. Streaming sends SSE chat.completion.chunk events then data: [DONE].
GET /v1/models: Lists the served model (owned_by: strata).
GET /health: Liveness + live state: policy, budget, resident_gb.

CORS is open; the server is batch-1 and unauthenticated — put it behind your own gateway beyond a trusted network.

CLI commands

strata bench: Fit + paged tok/s + fidelity + RAM for a model.
strata serve: OpenAI-compatible endpoint over the paged runtime.
strata optimize: Pick the best config within a quality budget; writes a deployment config.
strata report: Customer-readable HTML report from result JSONs.
strata status: Engine + artifact inventory (needs the source checkout).
strata profile / calibrate / tournament: Routing research (need the source checkout).

strata <command> --help for flags. Every result carries full provenance — see the methodology and the JSON schema.

Source-available under BSL 1.1 (becoming Apache 2.0 on 2030-08-04). Free for non-production use; production/commercial licensing: licensing@amperor.ai.