01 · INGEST — Corpus + Manifest

Ingest

The manifest-driven front door of the Nucleus pipeline: declare intent in superskill.yaml, drop files in a folder, and raw documents become overlapping, trust-tagged chunks.

● available now — gate-passed World corpora + the corpus_sha256 lockfile

Roadmap: the full ingestion gateway — file scanner, OCR, metadata tagger

bus · nucleus.ingestion.chunks.ready

The technical how — corpus in, manifest declared

Ingest is Stage 01 of the Nucleus pipeline — the point where declared intent meets raw material. Intent arrives as a superskill.yaml manifest, the build's contract: it declares the mode (1 = domain distillation, 2 = user-data-enriched, 3 = hybrid), the domain and its subdomains, the teacher and student models, and per-path source trust (primary/high, reference/medium, supplementary/low) — and the manifest schema is the production-grade core of the stage, with mode routing enforced in the orchestrator graph: Mode 1 skips Ingest entirely because the teacher already knows the domain, while Modes 2 and 3 must pass through it, and user files always land at the highest trust tier with no approval workflow. Material arrives as a drop folder handled by the Ingestion Gateway: a File Scanner walks the folder recursively and identifies every file by extension and content sniffing; format handlers extract text natively from text-bearing files (TXT/MD, PDF via pdfplumber, DOCX, CSV/XLSX, JSON, IPYNB, HTML/XML, source code, email archives), fall back to OCR (Tesseract) for scans and images, and convert PPT/EPUB/RTF first; a Metadata Tagger estimates dates, topics, language, and source confidence from content plus filenames; and a Chunker splits long documents into overlapping windows (~512–2048 tokens, configurable) so context that straddles a chunk boundary survives in both neighbors. The full scanner-OCR-tagger gateway is the stage's designed intake; the proven path running today chunks a single user document with overlap — the same chunk shape the gateway emits — and has fed real Mode 2 training runs. Finished chunks are announced on the NATS subject nucleus.ingestion.chunks.ready, handing the corpus to KICE+TICE for seven-layer extraction.

Declared intent — superskill.yaml

Nothing enters the pipeline undeclared. The manifest is the build's contract — mode, domain and subdomains, teacher and student, per-path source trust — and mode routing is enforced in the orchestrator graph, not by convention.

Mode 1Domain distillationskips Ingest entirely — the teacher already knows the domain

Mode 2User-data-enrichedpasses through Ingest — user files land at the highest trust tier, no approval workflow

Mode 3Hybridpasses through Ingest — declared sources plus user data

Per-path source trust: primary / high · reference / medium · supplementary / low.

The Ingestion Gateway — four moves

File Scanner

walks the drop folder recursively — every file identified by extension plus content sniffing

Text extraction / OCR

native handlers for text-bearing formats; OCR (Tesseract) for scans and images; PPT/EPUB/RTF converted first

Metadata Tagger

estimates dates, topics, language, and source confidence from content plus filenames

Chunker

overlapping windows (~512–2048 tokens, configurable) — context that straddles a boundary survives in both neighbors

The proven path running today chunks a single user document with overlap — the same chunk shape the gateway emits — and has fed real Mode 2 training runs. Roadmap: the full scanner → OCR → metadata-tagger gateway over arbitrary drop folders.

Inputs

Raw documents · OCR scans · superskill.yaml

Outputs

Chunked corpus · metadata-tagged shards

NATSfinished chunks announce on nucleus.ingestion.chunks.ready — handing the corpus to KICE + TICE for seven-layer extraction.

Worked exampleThe Tech Brief's farmer scenario is the stage's worked example: four generations of family farming notes — transcribed 1920s crop-rotation logs, Word-doc weather observations, scanned soil-test PDFs, markdown recipes, photos of field damage — with no structure and no metadata. The farmer fills out a minimal superskill.yaml (Mode 2; subdomains like crop_rotation and soil_management; family_notes/ as primary/high trust, county-extension pamphlets as reference/medium, photos as supplementary/low; era_range 1920–2026 so temporal conflicts can be contextualized downstream) and drops the files in raw. The gateway scans, extracts, OCRs the scanned PDFs, tags, and chunks with overlap — no data scientist required. Mid-training, the design adds hot-injection: qkz corpus add soil-analysis-2024.csv --subdomain soil-chemistry would fold new material in without a restart, and the next adversarial cycle picks it up.

The vocabulary

kice-tice

KICE + TICE

Stage 02 — twin agents mine the corpus in parallel across seven layers of knowledge, from rare concepts down to tribal know-how.

in the dictionary →

For agents

$ curl "https://qukaizen.com/what?term=ingest&world=nucleus"

# the raw compiled artifact: /worlds/nucleus/terms.json