ARCHITECTURE — THE ENGINE

Stream the model.
Don't hold it.

AeroLLM serves frontier-scale teachers (70B–400B+) on a single box by streaming layers from disk, executing on metal, and never requiring full-model residency. Speculative decoding gives 7× on Tier-1 teachers. On Apple Silicon, unified memory makes the storage→compute hop zero-copy.

View the explainer →
Layer streaming — NVMe shards → unified-memory window → MLX compute, KV cache quantized mid-stream

CAPACITY — WILL IT RUN?

Disk holds the model. RAM holds the working set.

Pick a model, quantization, and context length. Disk grows with total parameters — terabytes for a frontier teacher. AeroLLM's RAM stays bounded by the resident layer set, prefetch buffer, and KV cache. That decoupling is the whole pitch.

ON DISKLAYER STREAMAEROLLMIN MEMORY39.4 GB70B params · Q4whole model on diskLYRLYRLYRLYRLYRLYRprefetch ×8 (greedy)492 MB / layerAeroLLMstreaming engineexecute · evict · repeatLayer work: 1.5 GBPrefetch: 3.9 GBContext KV: 10.7 GBRuntime: 1.8 GB64G18.0 GBAeroLLM working setBREAKDOWNLayer work1.5 GBPrefetch3.9 GBContext KV10.7 GBRuntime1.8 GBTotal RAM18.0 GBvs 64 GB → fits with headroom

Model

Dense teacher class · projected on AeroLLM

Quantization

0.56 B/param · ~3.5× smaller

Context — 32K tokens

2K128K

Prefetch

Roadmap

Greedy grabs RAM for max throughput. Throttled stays out of the way — because agents wait, humans don't.

Will it run? — tap a machine to mark it on the bar

INTERFACE — RUN ANYTHING THAT FITS

If it fits on disk, it's an endpoint.

AeroLLM speaks an OpenAI-compatible API, so any compatible model that fits on your disk becomes a drop-in endpoint — the same live inference the AI Dictionary ships today, scaled up to models that never fit in RAM.

Serve which model

Quant

aerollm · chatIllustrative

> What is layer streaming, in one line?

▋

served by Qwen/Qwen2.5-7B-Instruct · streamed from disk
OpenAI-compatible
from openai import OpenAI

# point any OpenAI SDK at your AeroLLM box
client = OpenAI(base_url="http://localhost:8080/v1", api_key="aerollm")

stream = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Explain layer streaming."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

Base URL is your AeroLLM box — drop it into any OpenAI SDK.

Qwen2.5-7B≈ 4.3 GB on disk→streams in ≈ 3.9 GB working RAM (8K ctx, greedy)measured

Not hypothetical

The AI Dictionary already generates definitions live from an on-box model today. AeroLLM is the same streaming inference — extended to models too big to hold in RAM.

See it live →

Preview · In-browser inference through AeroLLM on this site is coming once the engine is wired in. The API shape above is real and OpenAI-compatible — today you run it yourself from the open-source engine.

STORAGEMEMORYAEROLLMCOMPUTETOKENS70B–400B Teacher · Layer-by-layerLAYER 00LAYER 01LAYER 02LAYER 03LAYER 04LAYER 05LAYER 06LAYER 07LAYER 08LAYER 09LAYER 10LAYER 11PAGE CACHEwarm layersUNIFIED MEMORYzero-copy · MLXKV CACHEattention reuseLAYER STREAMprefetch + dispatchAEROLLM COREstreaming engineSPECULATIVE DRAFTER1B helper · 7×VERIFIERparallel ratifyMLX / METALapple siliconCUDAdiscrete GPUCPU FALLBACKGGUF · llama.cppTOKEN STREAMOpenAI-compatible APIspec ⟳ verifyStream layer N from disk → execute on metal → token out — repeat for layer N+1Whole-model residency never required. Zero-copy on Apple Silicon. Speculative decoding gives 7× on Tier-1 teachers.

STORAGE

Layer-by-layer on disk

The 400B teacher sits on SSD as ~120 layer shards. AeroLLM prefetches the next layer while the current one runs. Whole-model residency never required — VRAM/unified-memory ceiling stops mattering.

AEROLLM CORE

Streaming + speculative

Layer dispatcher coordinates prefetch, page cache, and KV cache. A small drafter model proposes tokens; the verifier ratifies them in parallel against the teacher. Net: ~7× wall-clock speedup on tier-1 teachers, no quality loss.

COMPUTE

MLX, CUDA, or CPU

On Apple Silicon, MLX runs against unified memory — no host↔device copies, ~83% less power than discrete GPU. CUDA path takes the same layer stream over PCIe. CPU/GGUF fallback for boxes without an accelerator.

Dictionary →