ARCHITECTURE — THE ENGINE
Stream the model.
Don't hold it.
QueueLLM serves frontier-scale teachers (70B–400B+) on a single box by streaming layers from disk, executing on metal, and never requiring full-model residency. Speculative decoding gives 7× on Tier-1 teachers. On Apple Silicon, unified memory makes the storage→compute hop zero-copy.
CAPACITY — WILL IT RUN?
Disk holds the model. RAM holds the working set.
Pick a model, quantization, and context length. Disk grows with total parameters — terabytes for a frontier teacher. QueueLLM's RAM stays bounded by the resident layer set, prefetch buffer, and KV cache. That decoupling is the whole pitch.
Model
Dense teacher class · projected on QueueLLM
Quantization
0.56 B/param · ~3.5× smaller
Context — 32K tokens
Prefetch
RoadmapGreedy grabs RAM for max throughput. Throttled stays out of the way — because agents wait, humans don't.
Will it run? — tap a machine to mark it on the bar
INTERFACE — RUN ANYTHING THAT FITS
If it fits on disk, it's an endpoint.
QueueLLM speaks an OpenAI-compatible API, so any compatible model that fits on your disk becomes a drop-in endpoint — the same live inference the AI Dictionary ships today, scaled up to models that never fit in RAM.
Serve which model
Quant
> What is layer streaming, in one line?
▋
from openai import OpenAI
# point any OpenAI SDK at your QueueLLM box
client = OpenAI(base_url="http://localhost:8080/v1", api_key="aerollm")
stream = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=[{"role": "user", "content": "Explain layer streaming."}],
stream=True,
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="")Base URL is your QueueLLM box — drop it into any OpenAI SDK.
Not hypothetical
The AI Dictionary already generates definitions live from an on-box model today. QueueLLM is the same streaming inference — extended to models too big to hold in RAM.
Preview · In-browser inference through QueueLLM on this site is coming once the engine is wired in. The API shape above is real and OpenAI-compatible — today you run it yourself from the open-source engine.
NATIVE MOE — SELECTIVE EXPERT STREAMING
Only fetch the experts the router picked.
Mixture-of-Experts models route each token to a handful of experts per layer and skip the rest — Qwen3-30B-A3B activates 8 of 128. Naive streaming still reads every expert off disk regardless of routing. QueueLLM's AERO_MOE_SELECT mode streams only the active experts — same math, same tokens out (bit-exact, max_abs_diff == 0.0), just far less to read.
WHOLE-LAYER (DEFAULT)
0.216 tok/s
Reads all 128 experts per layer, forced streaming (ring_depth=4).
SELECTIVE (AERO_MOE_SELECT=1)
3.478 tok/s
Reads only the 8 active experts per layer — 16× faster, same output, same forced-streaming regime.
AERO_MOE_SELECT=1 aerollm generate \ --backend mlx-native --model Qwen3-30B-A3B-Instruct-2507-4bit \ --ring-depth 4 --prompt "..."
Measured on Qwen3-30B-A3B-Instruct-2507-4bit, forced streaming at ring_depth=4, 64 new tokens (phase0-glm52-killtest/run_aerollm_moe_efficiency.sh). Consistent with the 16× expert-I/O reduction documented in docs/streaming-economics.md. A bounded across-token expert cache was also tested and showed no additional win at this scale — measured, not shipped as a claim.
STORAGE
Layer-by-layer on disk
The 400B teacher sits on SSD as ~120 layer shards. QueueLLM prefetches the next layer while the current one runs. Whole-model residency never required — VRAM/unified-memory ceiling stops mattering.
QUEUELLM CORE
Streaming + speculative
Layer dispatcher coordinates prefetch, page cache, and KV cache. A small drafter model proposes tokens; the verifier ratifies them in parallel against the teacher. Net: ~7× wall-clock speedup on tier-1 teachers, no quality loss.
COMPUTE
MLX, CUDA, or CPU
On Apple Silicon, MLX runs against unified memory — no host↔device copies, ~83% less power than discrete GPU. CUDA path takes the same layer stream over PCIe. CPU/GGUF fallback for boxes without an accelerator.