frontier reasoning, streamed off your disk.
Don't shrink the model. Stream it.
A 235B model doesn't fit in your VRAM — so most tools shrink it until it does, and lose what made it smart. QueueLLM does the opposite: it keeps the model whole and feeds it through your GPU one layer at a time, streamed straight off your SSD.
Load a layer, compute, discard it, load the next. The full weight set never sits in memory at once — so frontier-scale reasoning runs on a laptop with no cluster and no cloud. With full credit to AirLLM for the layer-streaming insight; QueueLLM is that idea rebuilt in Rust for the stability and Apple Silicon (MLX) support a long pipeline run needs.
One question has driven most of the lab's work: how do we make inference faster and use fewer resources at the same time? Streaming alone trades speed for fit. The three primitives below — prefetch, speculate, shard the cache — are how QueueLLM claws the speed back without giving the memory budget back. Each has its own entry in the Performance cluster of the dictionary.
The whole model lives on your SSD. QueueLLM pulls it through a single small compute window — one layer resident at a time — and out the other side come frontier tokens. A 37GB Llama-3.1-70B checkpoint streams on a 36GB Mac at a measured 17.4GB peak — less memory than the model's own file on disk. And sparsity, not size, is the real axis: a 235B Mixture-of-Experts model (22B active per token) streams on the same machine at ~20GB peak.
Weights flow off the SSD layer by layer — the model is never fully resident.
The next layer loads while the current one computes, so the GPU rarely waits.
A small draft model proposes tokens; the big one verifies in a single pass.
One deterministic Rust binary — the inference backbone of the Nucleus pipeline.
Frontier scale, off your disk. Reasoning that compounds, on hardware you already own.
No Python runtime to crash, no fragile dependency tree. A single Rust executable with a deterministic lifecycle — start it, stream a model, shut it down clean.
One self-contained Rust executable. No interpreter, no venv — a deterministic lifecycle built to run for days without falling over.
Pulls weights off the SSD one transformer layer at a time. The full model never has to fit in VRAM.
Reads the next layer from disk while the current layer is still computing, hiding storage latency behind GPU work.
A fast draft model proposes a run of tokens; the big model verifies them in one pass. Lossless — same output, up to 7× the speed on 70B+ teachers.
DaC defines the vocabulary and compiles it once to terms.gbnf. QueueLLM loads that grammar once, then consults it at every single decode step — masking disallowed tokens at the logit level, inline with the forward pass, before sampling ever picks one.
Attention state is kept lean and sharded so long prompts fit alongside a streamed model instead of crowding it out.
Apple Silicon (MLX) is the production lane. The CUDA backend is underway and already certified for dense inference against an independent HuggingFace/PyTorch oracle — two separately written implementations agreeing token-for-token — with its streaming memory bound verified on an RTX 3090. Same engine, same API, swap with a flag.
Why it matters: a system prompt asking a model to
“only use sourced terms” is a request it can ignore — it is
still free to sample any token in its vocabulary, hallucination
included. A grammar is not a request. With terms.gbnf
loaded, an off-vocabulary token isn't discouraged — it's masked to
zero probability, unreachable rather than merely unlikely. The model
is structurally incapable of emitting a term DaC's gate never sourced.
Every token in the model's full vocabulary is reachable at every step. A plausible-sounding hallucinated entity or off-taxonomy claim can slip straight through — nothing at the decode level catches it.
Only DaC's gate-passed, sourced vocabulary is reachable at every step. A hallucinated term isn't rare — it's impossible to sample in the first place.
Why a state machine, not regex: the grammar only has to remember where it is — one pointer into the automaton — so checking the next token is a cheap lookup, not a re-scan. Regex validation run token-by-token has to re-match against the whole growing output every time, so it slows down the longer generation runs. A state machine doesn't care how long the output gets — checking token 100,000 costs exactly what checking token one did. The same reason state machines beat regex in production redaction pipelines is the reason they win here.
Rent the math, or own the run.
Measured, not promised: a 235-billion-parameter Mixture-of-Experts model streams from a laptop SSD in ~20GB of memory, and a 117B MoE produces coherent output on a 24GB machine. No rack of GPUs to rent, no per-token meter running, no data leaving the machine.
For Mixture-of-Experts models the engine goes further: out of the box it streams only the experts each token actually routes to — a measured 16× cut in expert I/O versus loading whole layers, with bit-identical output. Streaming off disk trades latency for reach, so it opens doors that real-time chat never could: research, batch reasoning, evaluation, offline pipelines — anywhere answers aren't response-time-sensitive. A deep model that stays up, costs nothing per query once it's on disk, and answers to no one but you.
total parameters, measured streaming on a 36GB Mac at ~20GB peak — because only the 22B active parameters per token set the pace. Dense 70B: 17.4GB peak, below its own checkpoint size.
Who else streams a 235-billion-parameter model on a laptop?
Clone the engine, point it at a model, and stream 400B+ parameters off the disk you already own.