frontier reasoning, streamed off your disk.
Don't shrink the model. Stream it.
A 671B model doesn't fit in your VRAM — so most tools shrink it until it does, and lose what made it smart. QueueLLM does the opposite: it keeps the model whole and feeds it through your GPU one layer at a time, streamed straight off your SSD.
Load a layer, compute, discard it, load the next. The full weight set never sits in memory at once — so frontier-scale reasoning runs on a laptop with no cluster and no cloud. With full credit to AirLLM for the layer-streaming insight; QueueLLM is that idea rebuilt in Rust for the stability and Apple Silicon (MLX) support a long pipeline run needs.
One question has driven most of the lab's work: how do we make inference faster and use fewer resources at the same time? Streaming alone trades speed for fit. The three primitives below — prefetch, speculate, shard the cache — are how QueueLLM claws the speed back without giving the memory budget back. Each has its own entry in the Performance cluster of the dictionary.
The whole model lives on your SSD. QueueLLM pulls it through a single small compute window — one layer resident at a time — and out the other side come frontier tokens. A Mac that could never hold a 405B model in memory can now run one — measured at a 12GB peak resident set.
Weights flow off the SSD layer by layer — the model is never fully resident.
The next layer loads while the current one computes, so the GPU rarely waits.
A small draft model proposes tokens; the big one verifies in a single pass.
One deterministic Rust binary — the inference backbone of the Nucleus pipeline.
Frontier scale, off your disk. Reasoning that compounds, on hardware you already own.
No Python runtime to crash, no fragile dependency tree. A single Rust executable with a deterministic lifecycle — start it, stream a model, shut it down clean.
One self-contained Rust executable. No interpreter, no venv — a deterministic lifecycle built to run for days without falling over.
Pulls weights off the SSD one transformer layer at a time. The full model never has to fit in VRAM.
Reads the next layer from disk while the current layer is still computing, hiding storage latency behind GPU work.
A fast draft model proposes a run of tokens; the big model verifies them in one pass. Lossless — same output, up to 7× the speed on 70B+ teachers.
DaC defines the vocabulary and compiles it once to terms.gbnf. QueueLLM loads that grammar once, then consults it at every single decode step — masking disallowed tokens at the logit level, inline with the forward pass, before sampling ever picks one.
Attention state is kept lean and sharded so long prompts fit alongside a streamed model instead of crowding it out.
Apple Silicon unified memory (zero-copy, no passthrough) or a discrete NVIDIA GPU — same engine, same API, swap with a flag.
Why it matters: a system prompt asking a model to
“only use sourced terms” is a request it can ignore — it is
still free to sample any token in its vocabulary, hallucination
included. A grammar is not a request. With terms.gbnf
loaded, an off-vocabulary token isn't discouraged — it's masked to
zero probability, unreachable rather than merely unlikely. The model
is structurally incapable of emitting a term DaC's gate never sourced.
Every token in the model's full vocabulary is reachable at every step. A plausible-sounding hallucinated entity or off-taxonomy claim can slip straight through — nothing at the decode level catches it.
Only DaC's gate-passed, sourced vocabulary is reachable at every step. A hallucinated term isn't rare — it's impossible to sample in the first place.
Why a state machine, not regex: the grammar only has to remember where it is — one pointer into the automaton — so checking the next token is a cheap lookup, not a re-scan. Regex validation run token-by-token has to re-match against the whole growing output every time, so it slows down the longer generation runs. A state machine doesn't care how long the output gets — checking token 100,000 costs exactly what checking token one did. The same reason state machines beat regex in production redaction pipelines is the reason they win here.
Rent the math, or own the run.
The deepest open models there are — up to 671B parameters at 4-bit — run locally, streamed from a 1TB laptop SSD. No rack of GPUs to rent, no per-token meter running, no data leaving the machine.
Streaming off disk trades latency for reach, so it opens doors that real-time chat never could: research, batch reasoning, evaluation, offline pipelines — anywhere answers aren't response-time-sensitive. A deep model that stays up, costs nothing per query once it's on disk, and answers to no one but you.
parameters at 4-bit — streamed from an SSD, run on commodity hardware. 405B measured with a 12GB peak resident set on a 36GB Mac.
Who else runs a 671-billion-parameter model on a laptop?
Clone the engine, point it at a model, and stream 400B+ parameters off the disk you already own.