The World Models Dictionary.

A variational framework where perception and action both minimize expected free energy, unifying curiosity and goal-seeking.

Friston's active inference framework: an agent maintains a generative model of the world and acts to minimize expected free energy, which decomposes into epistemic value (expected information gain — curiosity) and pragmatic value (expected reward / goal proximity). Unlike ICM or WorldLLM, curiosity emerges from the variational objective rather than being a separately engineered signal. Active inference is the mechanism underlying the active-inference-agent system.

An agent that minimizes expected free energy, balancing epistemic value (information gain) and pragmatic value (goal achievement).

An agent architecture based on the free-energy principle (Friston): actions are selected to minimize expected free energy, which decomposes into epistemic value (expected information gain — curiosity) and pragmatic value (expected reward / goal proximity). Active inference unifies perception, action, and curiosity under one variational objective. It differs from ICM/WorldLLM in deriving curiosity from the variational free-energy objective rather than from a prediction-error or log-likelihood signal.

Updating a probability distribution over hypotheses (the posterior) when new evidence arrives, using Bayes' rule.

The general mechanism: Bayes' rule combines a prior over hypotheses P(H) with a likelihood P(evidence | H) to produce a posterior P(H | evidence). Bayesian inference is the general anchor for both Bayesian surprise (the signal) and the Theorist (which performs Bayesian NL hypothesis refinement). It is NOT a learning algorithm — it is the mathematical framework underlying both active inference and the WorldLLM theory-update step.

The KL divergence between the agent's posterior over models before and after an observation.

A *different instance* of intrinsic motivation from prediction-error curiosity: surprise measured as how much an observation moves the agent's posterior over models (KL[posterior‖prior]) — i.e. information gained about the world's structure. Because it rewards posterior *change*, not raw unpredictability, it avoids the noisy-TV trap. In WorldLLM this is the signal the Experimenter maximizes — it seeks transitions the current theory finds surprising (low log-likelihood). It is contrasted with prediction-error curiosity: same word 'surprise', different math.

A benchmark for automated experimental design and model discovery, where an agent proposes experiments and updates a model of the world.

Gandhi et al.'s BoxingGym (arXiv:2501.01540): a benchmark environment for evaluating automated experimental design and theory-making. An agent proposes experiments, observes outcomes, and updates a probabilistic model of the world — a formal test bed for hypothesis-refinement machinery like the WorldLLM Theorist loop. It operationalizes the scientist-as-agent framing and provides a quantitative evaluation framework for curiosity-driven model discovery.

A curiosity agent finds ways to maximize its intrinsic reward without encountering genuinely novel or informative states.

A failure mode distinct from the noisy-TV problem: the agent learns to maximize the curiosity signal through adversarial exploitation of the feature space or model artifacts rather than by genuinely exploring novel states. For example, in ICM the agent may find feature-space transitions that are high-error without corresponding to novel environment states. It is a generalization of reward hacking to intrinsic-reward systems.

A latent world-model agent that learns an environment model in a compact learned embedding space and trains its policy on imagined rollouts.

Hafner et al.'s DreamerV3 (arXiv:2301.04104, building on arXiv:1803.10122): a model-based RL agent that learns a Recurrent State Space Model (RSSM) as a latent world model and trains a policy entirely on imagined trajectories within it. The world model is a latent embedding, not a natural-language theory — which is the key contrast with WorldLLM's Theorist. Dreamer uses extrinsic reward and does not use curiosity-driven exploration by default; it enters this World as a world-model predictor, not as a curiosity system.

Epistemic uncertainty is reducible by gathering data; aleatoric uncertainty is irreducible noise in the environment.

A fundamental distinction for curious agents: epistemic (model) uncertainty arises from insufficient data — it can be reduced by exploring. Aleatoric (environment) uncertainty is inherent randomness that cannot be explained by any model — exploring it yields no new knowledge. A good curiosity signal rewards exploration of epistemic uncertainty and ignores aleatoric uncertainty. The noisy-TV problem arises when prediction-error curiosity cannot distinguish the two: a random TV is maximally aleatoric but generates infinite prediction error.

WorldLLM's curiosity-driven RL agent that seeks transitions the current theory finds surprising.

One realization of curiosity-driven / intrinsic-motivation exploration (its general anchor): a policy trained to reach environment transitions to which the Theorist's current natural-language theory assigns LOW log-likelihood (high Bayesian surprise). It generates the data that forces the theory to improve. It is the 'curiosity' half of the WorldLLM loop and an instance of intrinsic-motivation exploration — NOT a general RL control policy and NOT RLHF.

The strategy of visiting unknown or uncertain states to gather information, trading off against exploitation of known rewards.

The exploration/exploitation trade-off is a foundational problem in RL: an agent must sometimes choose actions that gather information (exploration) rather than actions that maximize known reward (exploitation). Intrinsic motivation is one family of exploration strategies — rewarding the agent for visiting novel or surprising states. Other families include epsilon-greedy, UCB, and Thompson sampling. Exploration is the general category; curiosity-driven exploration is a specific instance.

The variational objective in active inference: minimizing the gap between the agent's model and the true world.

In the free-energy principle, the variational free energy F = KL[Q(world) || P(world | obs)] + H[P(obs | model)] is the objective agents minimize. Minimizing F achieves two things simultaneously: (1) updating the agent's beliefs toward the evidence (perception / Bayesian posterior update), and (2) acting to make the evidence match the prior (goal-seeking). Epistemic free energy (the curiosity component) is the expected F reduction from an action — how much the action would reduce the agent's uncertainty.

Iteratively revising a hypothesis (theory) to better explain observed evidence, using likelihood scoring.

The general process of updating a working hypothesis to increase its likelihood given observed evidence — analogous to Bayesian posterior update but not always fully Bayesian. The WorldLLM Theorist is one realization: it refines a natural-language theory of environment dynamics by scoring candidate theories by their likelihood on collected transitions and proposing revisions via the LLM's in-context reasoning. BoxingGym formalizes this as automated experimental design.

A non-generative latent world model that predicts abstract representations of image regions, not pixel-level reconstructions.

Assran et al.'s I-JEPA (arXiv:2301.08243): a self-supervised learning architecture that predicts the abstract representation of target image regions from context regions in latent space — a latent predictive world model that does not reconstruct pixels. The key contrast with generative models (VAE/diffusion) is that JEPA predicts in embedding space, avoiding the wasted capacity of pixel-level generation. It enters this World as a world-model predictor (a latent predictive architecture), not as a curiosity or planning system.

A self-supervised curiosity system where intrinsic reward = forward model prediction error in a learned feature space.

Pathak et al.'s Intrinsic Curiosity Module (arXiv:1705.05363): a self-supervised system that learns a feature space of states via an inverse model (predicting actions from state transitions) and trains a forward model in that space. Intrinsic reward = the forward model's prediction error. ICM is the canonical instance of prediction-error curiosity and the system that introduced the noisy-TV problem as a named failure mode. It contrasts with Bayesian-surprise-based systems: ICM rewards raw prediction error; Bayesian surprise rewards posterior change.

An LLM's ability to update its behavior based on examples or evidence provided in the prompt, without weight updates.

The capacity of large language models to adapt their outputs to new evidence or examples provided in the context window, without any gradient-based parameter update. In the WorldLLM Theorist, in-context learning is the substrate for hypothesis refinement: the Theorist updates its theory by conditioning on new transitions in the context, not by fine-tuning. This makes the Theorist's 'Bayesian update' an in-context, not a gradient-based, operation.

The expected reduction in entropy of the agent's belief over hypotheses, given an action/observation.

The expected KL divergence between the posterior and prior over world models or hypotheses after taking an action and observing the outcome. Information gain formalizes 'how much will I learn from this action?' and is the objective behind optimal Bayesian experimental design. It is a generalization of Bayesian surprise to expectations over possible outcomes. Active-inference agents maximize expected information gain (epistemic value) as part of their free-energy objective.

Reward an agent generates internally to drive exploration, absent external task reward.

A self-generated drive to explore — the agent rewards itself for novelty, surprise, or learning progress rather than for an extrinsic task signal. It is the umbrella concept; prediction-error curiosity and Bayesian surprise are two distinct *instances* of it (a distinction this World exists to make clear). Sutton & Barto treat it as exploration shaping; it is NOT a control algorithm and NOT RLHF.

An agent exploits the curiosity signal itself as a reward to hack rather than as a guide to genuine exploration.

The general failure mode where an agent finds policies that maximize the intrinsic curiosity reward without achieving the intended effect (exploring novel, informative states). This includes the noisy-TV problem (aleatoric trap), detachment (feature-space exploitation), and count-based hacking (cycling through rarely-visited but uninformative states). It is the intrinsic-reward analog of extrinsic reward hacking. The fix is a curiosity signal that rewards epistemic value, not raw prediction error or visit rarity.

Surprise measured as the negative log-probability a model assigns to an observed outcome: -log P(observation | model).

The basic information-theoretic definition of surprise: -log P(x | model). High log-likelihood surprise means the current model finds the observation improbable. WorldLLM's Experimenter uses this as its curiosity signal — it seeks transitions the Theorist's current NL theory assigns low log-likelihood (high log-likelihood surprise). It is a special case of Bayesian surprise when the prior and posterior are collapsed to the current model's point estimate.

Model-based agents plan using an explicit world model; model-free agents learn a policy or value function directly from experience.

The primary architectural split in RL: model-free agents (Q-learning, policy gradients) learn to act by directly optimizing from environment rewards without an explicit environment model — they require many real-environment samples. Model-based agents (Dyna, Dreamer, WorldLLM) maintain an explicit world model and can plan, simulate, or reason about actions before executing them — more sample-efficient but require a world model that is accurate enough. Curiosity systems can be attached to either, but model-based agents have more natural access to the transition model that curiosity signals need.

Prediction-error curiosity gets trapped by inherently unpredictable (aleatoric) stimuli.

A failure mode of prediction-error curiosity: an agent rewarded for prediction error fixates on stochastic, unlearnable stimuli (a TV showing static) because prediction error never falls. It is why distinguishing EPISTEMIC uncertainty (reducible — worth exploring) from ALEATORIC uncertainty (irreducible noise — a trap) matters, and a key argument for Bayesian-surprise / information-gain signals that reward *model improvement* rather than raw unpredictability.

An intrinsic reward proportional to how rarely a state or observation has been visited.

A count-based or density-model intrinsic reward: the agent is rewarded for visiting states or observations it has encountered rarely (low visit count or low density under a learned model). Novelty signals are the simplest instance of intrinsic motivation and predate ICM. They do not require a forward model or posterior over theories — only a count or density estimate. They suffer from the same noisy-TV vulnerability if random states have low visit count but are uninformative.

An MDP where the agent does not directly observe the full environment state — it infers a belief state.

A Partially Observable Markov Decision Process extends the standard MDP with a hidden state space: the agent receives observations rather than full state information and must maintain a belief distribution (a posterior over possible states). World models are especially valuable in POMDPs because the model can help the agent infer the latent state from partial observations. Curiosity signals in POMDPs reward exploration of regions of state-space where the belief distribution is uncertain.

Intrinsic reward = how wrong the agent's forward model is about the next state.

One *instance* of intrinsic motivation (Pathak's ICM): the agent's intrinsic reward is the error of a learned forward dynamics model in a self-supervised feature space. High prediction error = high curiosity. It CONTRASTS WITH Bayesian surprise: prediction-error rewards being wrong about the next state; Bayesian surprise rewards the *change in the posterior over models*. Conflating the two is the exact confusion this term resolves. Its known pathology is the noisy-TV problem.

A distribution from which candidate hypotheses or actions are sampled, then accepted/rejected or scored.

In Bayesian inference and MCMC, the proposal distribution Q(H' | H) generates candidate hypotheses H' given the current hypothesis H; a scoring or acceptance step decides whether to adopt H'. The WorldLLM Theorist uses the LLM itself as the proposal distribution over NL theory revisions — the LLM proposes candidate theories conditioned on collected observations and the current theory, and the theory with highest transition likelihood is adopted. This is the technical sense in which the Theorist is 'Bayesian hypothesis refinement with an LLM as the proposal distribution.'

[BUILT] QuKaiZen's assertClosedSourcedGraph gate: the mechanism that keeps autonomous hypotheses RAW until sourced evidence supports them.

[BUILT] The assertClosedSourcedGraph function (src/gate.ts, QuKaiZen DaC engine) is the provenance gate: it enforces three laws — (1) every term carries a non-empty source, (2) every category is declared in the spec, (3) every related edge resolves to a compiled term. In the context of a WorldLLM-style Theorist/Experimenter loop, the gate is the reality-anchor: a Theorist hypothesis stays RAW (uncompiled) until the gate confirms it is sourced to verifiable evidence. This is BUILT today; the autonomous loop running on it is ROADMAP (see reality-anchored-theorist).

[ROADMAP] QuKaiZen's intended Theorist/Experimenter loop, safety-anchored by the provenance gate.

[ROADMAP] The QuKaiZen-internal application of the WorldLLM Theorist/Experimenter loop (arXiv:2506.06725): an autonomous curiosity-driven loop in which hypotheses generated by the Theorist remain RAW until the provenance gate (assertClosedSourcedGraph — BUILT today in src/gate.ts) confirms sourced evidence supports them. This loop does not yet run end-to-end; it is the autonomy ROADMAP. The gate (BUILT) is the reality-anchor; the loop running on it is ROADMAP. See provenance-gate-anchor for the BUILT component.

An observed transition the current theory rates as improbable — the WorldLLM curiosity signal.

The concrete signal the Experimenter maximizes: a state→action→next-state transition assigned low log-likelihood under the Theorist's current theory. It is WorldLLM's operationalization of Bayesian surprise as an LLM log-likelihood, and the bridge from the curiosity signal to the theory update. It contrasts with raw prediction-error curiosity (ICM): both seek 'surprising' transitions, but log-likelihood surprise rewards model-inconsistency in natural language, while ICM rewards forward-model error in feature space.

WorldLLM's 2nd LLM that refines a natural-language theory of environment dynamics via Bayesian updating.

One realization of Bayesian hypothesis refinement with an LLM as the proposal distribution (its general anchor). The Theorist proposes/revises a natural-language theory of the environment's dynamics, scored by how well it predicts collected transitions (likelihood); surprising transitions from the Experimenter drive the update. It is the 'world-model' half of the loop — the world model here is an NL theory, NOT a JEPA latent predictor (a distinction this World makes explicit). The Theorist does NOT execute actions; the Experimenter does.

The component of a world model that predicts the next state given current state and action: P(s' | s, a).

The transition model (or dynamics model) specifies P(s' | s, a) — the probability distribution over next states given the current state and action. It is the core of any world model. Model-based RL agents learn a transition model from data and use it for planning or curiosity. ICM's forward model is a learned transition model in feature space; DreamerV3's RSSM is a learned stochastic transition model in latent space.

A Theorist-style agent's theory drifts away from reality because unverified hypotheses are promoted to truth without sourced evidence.

[ROADMAP] A failure mode specific to Theorist/Experimenter-style loops: if the Theorist's proposed hypotheses are promoted to the agent's 'compiled' world model without requiring sourced, evidence-backed verification, the theory drifts from reality — the agent acts on a model that was never grounded. In WorldLLM this is prevented by the scoring mechanism (only theories that predict transitions well survive). In QuKaiZen's provenance gate (BUILT — assertClosedSourcedGraph), it is prevented at compile time by requiring every term to have a non-empty, traceable source. The QuKaiZen loop (ROADMAP) will need this gate as its reality-anchor; the gate itself is BUILT today.

The refined NL theory the loop converges toward — WorldLLM's improving world model.

The output of one WorldLLM loop iteration: a revised natural-language theory with higher likelihood on observed transitions, which then re-defines what the Experimenter finds surprising. Iterating closes the loop (propose → seek surprise → refine). It is the 'updated world model' in NL form. It contrasts with a latent world model (JEPA/Dreamer): those encode dynamics as learned embeddings; the updated theory encodes them as natural language propositions that a human could read.

An internal model of environment dynamics that an agent uses to predict outcomes of actions.

A learned or hand-coded model of an environment's transition dynamics — how states evolve under actions. World models enable model-based planning (imagined rollouts) without real environment interaction. The concept is general; different systems instantiate it differently: DreamerV3 uses a latent RSSM; I-JEPA uses a latent predictive embedding; WorldLLM uses a natural-language theory. This disambiguation is the core value of this World for the applied AI builder.

A curiosity-driven world-modeling system pairing an Experimenter (RL) with a Theorist (LLM) that refines a natural-language theory of environment dynamics.

A world-modeling system (arXiv:2506.06725) in which two LLMs collaborate: an Experimenter (curiosity-driven RL policy) that seeks low-log-likelihood transitions, and a Theorist that refines a natural-language theory of the environment's dynamics as a Bayesian hypothesis-refinement process. The world model here is a natural-language theory, not a latent embedding — which distinguishes it from Dreamer (latent) and JEPA (latent predictive). The system operationalizes Bayesian surprise as LLM log-likelihood and hypothesis refinement as in-context Bayesian update.