The Neural Networks Dictionary.

The element-wise nonlinearity applied after the weighted sum; determines what inputs a neuron responds to and enables the network to represent non-linear functions.

An activation function applies a fixed, element-wise nonlinear transformation to the pre-activation (weighted sum + bias) of each neuron: a = activation(z). Without nonlinearity, any stack of linear layers collapses to a single linear transformation — the activation is what makes depth meaningful. Key choices: sigmoid σ(z) = 1/(1+e^{-z}), squashes to (0,1), saturates in tails (vanishing gradient risk); tanh(z), zero-centered sigmoid, same saturation risk; ReLU max(0,z), the modern default — simple, gradient-preserving for z>0 but causes dead neurons for z≤0 (dead-relu problem); softmax (output layer for classification), converts logits to a probability distribution. The choice of activation directly determines gradient flow: ReLU's linear-for-positive region is why it enables deep networks that sigmoid cannot.

A learned weighting over a set of context vectors (values) indexed by similarity of a query to keys; enables the network to selectively focus on relevant parts of the input.

An attention mechanism computes a weighted sum of a set of value vectors, where the weights are derived from the similarity of a query vector to a set of key vectors: Attention(Q, K, V) = softmax(QKᵀ / √d_k) V. The query, key, and value vectors are linear projections of the input representation. The division by √d_k prevents the dot products from growing too large before the softmax (which would saturate gradients). Attention is the core operation of the transformer: it replaces the local receptive field of convolutions with a global, dynamically computed dependency between any two positions. Unlike RNN hidden states, attention creates direct, differentiable paths between any positions in the input — this is why transformers can model long-range dependencies better than recurrent networks in practice.

The systematic, exact computation of derivatives by applying the chain rule to a recorded computation graph — the engine behind backpropagation in modern frameworks.

Automatic differentiation (autodiff) is the algorithmic technique used by deep learning frameworks (PyTorch autograd, JAX, TensorFlow) to compute exact derivatives of any differentiable function defined by a computation graph. Unlike numerical differentiation (finite differences) or symbolic differentiation (expression manipulation), autodiff records the sequence of elementary operations in a 'tape' or computation graph during the forward pass, then applies the chain rule backward through this graph (reverse-mode autodiff) to produce exact gradients. Backpropagation, as defined in the neural network literature, is the instance of reverse-mode autodiff applied to neural network loss functions. Autodiff is not approximate (within floating-point precision) and scales to millions of parameters — it is what makes large neural networks trainable.

The algorithm for computing gradients of the loss with respect to all network weights by applying the chain rule backward through the computation graph.

Backpropagation (Rumelhart, Hinton & Williams, 1986) is the algorithm that makes training deep networks tractable. After the forward pass produces a loss value, backpropagation computes ∂L/∂w for every weight w by applying the chain rule from the output layer backward to the input layer. The key insight: the gradient of the loss with respect to an earlier layer's weights can be expressed as the product of the gradient from the layer above and the local Jacobian — so the computation flows backward through the same graph that the forward pass traverses, reusing stored intermediate activations. Backpropagation does not perform a weight update; it computes gradients that the optimizer then uses. In modern autodiff frameworks (PyTorch autograd, JAX), backpropagation is implemented as reverse-mode automatic differentiation.

A learned scalar added to the weighted sum before the activation; shifts the activation threshold independently of the input.

The bias is a scalar parameter added to the weighted sum of inputs before the activation function is applied: z = w · x + b. It gives the neuron the freedom to shift its activation function along the input axis — without a bias, the hyperplane the neuron defines must pass through the origin, severely limiting what functions the network can represent. Biases are learned parameters just like weights; backpropagation computes their gradient and gradient descent updates them. Conceptually, the bias can be absorbed into the weight vector by appending a constant 1 to the input, though in practice they are stored separately.

A neural network that applies learned convolutional filters across an input, exploiting spatial locality and weight sharing for translation-equivariant feature detection.

A Convolutional Neural Network (CNN) replaces the dense weight matrix of an MLP hidden layer with a set of learned convolutional filters (kernels) that are applied at every position across the input. This achieves two properties: locality (each filter sees only a small receptive field, detecting local features) and weight sharing (the same filter is used everywhere, dramatically reducing parameters and encoding translation equivariance). Introduced by LeCun et al. (1998) in the LeNet architecture for digit recognition, CNNs became the dominant architecture for computer vision tasks through AlexNet (2012) and subsequent deeper variants. A CNN is an MLP with a structured sparsity and weight-tying pattern in the hidden layers — the substrate is the same weighted-sum + activation-function unit, just arranged with spatial structure.

The failure mode where a ReLU neuron's pre-activation is always ≤ 0 during training, causing zero gradient and preventing the neuron from ever learning.

A ReLU neuron is 'dead' when its pre-activation (weighted sum + bias) is ≤ 0 for all inputs in the training set — because ReLU(z) = 0 for z ≤ 0 and the gradient is also 0 for z ≤ 0, backpropagation produces zero gradient for the neuron's weights, and they never update. Once a neuron is dead it stays dead: with no gradient, no update moves it back to a positive region. Dead neurons can be caused by too-large learning rates (a big weight update can push many pre-activations permanently negative), poor weight initialization (biases initialized too negative), or unlucky mini-batches. Mitigations: Leaky ReLU (gradient = small ε for z ≤ 0); ELU; careful learning rate scheduling; Kaiming initialization. Dead neurons represent wasted capacity — in a network with 30% dead neurons, those parameters contribute nothing to the learned function.

A regularization technique that randomly sets neuron activations to zero during training, preventing co-adaptation and reducing overfitting.

Dropout (Srivastava et al., JMLR 2014) is a regularization technique for neural networks: during each training forward pass, each activation unit is independently set to zero with probability p (the dropout rate), and the remaining activations are scaled by 1/(1-p) to maintain expected activation magnitude. This prevents neurons from co-adapting — relying on specific other neurons always being present — forcing each neuron to learn more robust, independent features. At inference time, dropout is disabled (all neurons active). Dropout is equivalent to averaging over an exponential number of thinned networks with shared weights, giving it an ensemble interpretation. It is most effective in large, overparameterized networks; in batch-normalized transformer models it is often applied at a lower rate or omitted because normalization provides its own regularization.

A learned dense vector representation of a discrete token (word, sub-word, image patch) in a continuous real-valued space; the standard interface between discrete inputs and neural networks.

An embedding layer maps a discrete symbol (a vocabulary index, a categorical value) to a dense real-valued vector via a learned embedding matrix E of shape [vocab_size × d_model]. It is a parameterized lookup table: the embedding for token i is the i-th row of E. Embeddings transform discrete, non-metric inputs into a metric space where semantic similarity corresponds to vector proximity, enabling gradient flow from the loss all the way back into the input representation. In transformers, input token embeddings are summed with positional embeddings before entering the attention layers — the positional embedding component gives the model the sequence order that self-attention alone cannot infer. Embeddings are the first layer of virtually every language model and the interface between symbolic inputs and the continuous neural network substrate.

The failure mode where gradients grow exponentially during backpropagation, causing weight updates to be catastrophically large and training to diverge.

The exploding gradient problem is the symmetric counterpart of the vanishing gradient problem: when Jacobian eigenvalues are > 1, gradients grow exponentially as they propagate backward through many layers. This leads to weight updates that are orders of magnitude larger than the weights themselves — the optimizer takes a catastrophically large step, the loss spikes, and training diverges. Exploding gradients are especially common in recurrent networks (RNNs/LSTMs) because the same weight matrix is multiplied repeatedly across time steps. The standard mitigation is gradient clipping: if the L2 norm of the gradient vector exceeds a threshold, rescale it to that threshold before the optimizer step. Residual connections also help by providing a bounded gradient path. Unlike vanishing gradients (which are silent and slow), exploding gradients manifest as NaN losses or sudden loss spikes — they are detectable but disruptive.

The computation that propagates an input through a network layer by layer to produce an output (prediction or score).

The forward pass (or forward propagation) is the process by which an input vector is transformed into a network output by applying each layer's computation in sequence: for each layer, compute the weighted sum of the incoming activations (linear step), add the bias, then apply the activation function (nonlinear step). The final layer produces the network's output — a class score, a probability, a regression value, or a sequence element. Crucially, the forward pass is also the computation that must be recorded (in an autodiff system) to enable backpropagation: every intermediate activation is stored so gradients can flow backward. In inference the intermediate activations are discarded to save memory.

The optimization algorithm that iteratively updates network weights in the direction of steepest loss decrease (negative gradient).

Gradient descent is the workhorse optimization algorithm for training neural networks. At each step, the gradient of the loss with respect to all weights (computed by backpropagation) is used to update the weights in the direction that most rapidly reduces the loss: w ← w - η·∂L/∂w, where η is the learning rate. Batch gradient descent uses all training examples to compute one gradient; stochastic gradient descent (SGD) uses a single example; mini-batch gradient descent (the standard in practice) uses a small random subset. The loss landscape of a deep network is non-convex with saddle points and local minima — but empirical evidence shows that gradient descent plus momentum (Adam, SGD+momentum) reliably finds solutions that generalize well. Gradient descent does not guarantee a global minimum, only convergence to a local stationary point.

The change in the distribution of each layer's inputs during training as preceding layer weights update; destabilizes learning and motivated batch normalization.

Internal covariate shift (Ioffe & Szegedy, arXiv:1502.03167) is the phenomenon where the distribution of each hidden layer's inputs changes continuously during training as the parameters of the preceding layers are updated. Because each layer is trained assuming a stable input distribution, this instability forces each layer to continuously re-adapt to its shifting inputs, slowing learning and requiring lower learning rates. Batch normalization was proposed specifically to reduce internal covariate shift: by normalizing layer inputs to zero mean and unit variance (over the mini-batch), the distribution presented to each layer is stabilized regardless of upstream parameter changes. The term is primarily associated with the batch normalization paper — subsequent research has debated whether batch normalization works specifically because it reduces internal covariate shift or through other mechanisms (smoothing the loss landscape).

A group of neurons processing inputs in parallel at the same stage of the network; the unit of composition in all deep architectures.

A layer is a group of neurons that collectively transform the same input into the same output — processing is parallel within a layer, sequential across layers. In a fully connected (dense) layer, every neuron receives every input; in a convolutional layer, neurons share weights and see only a local receptive field; in an attention layer, neurons attend over all positions via similarity scores. Layers are the unit of composition: depth (number of layers) determines the hierarchy of representations the network can build. A deeper network can represent more complex functions than a shallow one for the same total parameter count — this empirical observation underlies the preference for depth over width in modern architectures.

A scalar function measuring the discrepancy between the network's output and the desired target; the quantity backpropagation differentiates.

The loss function (also called objective, cost, or criterion) maps the network's output and the ground-truth label to a non-negative scalar that measures how wrong the prediction is. It is the quantity that gradient descent minimizes and that backpropagation differentiates. Common choices: cross-entropy loss for classification (measures divergence between predicted probabilities and one-hot labels); mean squared error (MSE) for regression; KL divergence for generative models. The choice of loss encodes the assumptions about the error distribution — cross-entropy assumes categorical output; MSE assumes Gaussian noise. A poor loss choice (e.g., MSE for class probabilities) leads to slow convergence or mode collapse regardless of the architecture.

A recurrent unit with gating mechanisms (input, forget, output gates) that selectively retain or discard information over long sequences, mitigating the vanishing-gradient problem.

The Long Short-Term Memory (LSTM, Hochreiter & Schmidhuber, 1997) replaces the simple RNN recurrence with a cell state and three multiplicative gates: the forget gate (what to erase from the cell), the input gate (what new information to write), and the output gate (what to expose as the hidden state). The gates are sigmoid-activated, producing values in [0,1] that multiply the cell state — this additive cell-state update path allows gradients to flow backward over hundreds of steps without vanishing. LSTMs were the dominant sequence model for NLP (translation, speech) before the transformer. They are the first architecture designed explicitly to address the vanishing-gradient failure mode of plain RNNs.

A feedforward neural network with one or more hidden layers of neurons; the foundational deep architecture from which all others derive.

A Multi-Layer Perceptron (MLP) is the canonical feedforward neural network: an input layer, one or more hidden layers of neurons, and an output layer, connected sequentially. Each layer computes a linear transformation (weighted sum + bias) followed by a nonlinear activation — without nonlinearity, stacked layers collapse to a single linear map. The MLP is trained end-to-end by backpropagation and gradient descent. It is the conceptual predecessor and substrate of all deeper architectures: CNNs add weight sharing and locality to hidden layers; RNNs add recurrence; transformers replace dense connectivity with attention. The universal approximation theorem (Hornik et al., 1989) proves that a sufficiently wide single-hidden-layer MLP can approximate any continuous function, but depth rather than width is the practical route to efficiency.

The richness of the function class a network can represent; determined by architecture depth, width, and activation choice — more capacity enables fitting more complex patterns but increases overfitting risk.

Capacity (also called expressivity) refers to the set of functions a neural network architecture can represent. A model with low capacity cannot fit complex training data (underfitting); a model with high capacity can fit complex data but may fit noise rather than the true pattern (overfitting). For neural networks, capacity is controlled by depth (number of layers), width (number of units per layer), and activation nonlinearity (ReLU increases effective capacity vs. polynomial activations). Classical bias-variance theory predicts that capacity should be tuned to match data complexity, but modern deep learning overturns this: massively overparameterized models (far more parameters than training examples) generalize well in practice (the 'double descent' phenomenon). Capacity is a property of the model class, not a single network — the universal approximation theorem shows MLPs have universal capacity in the limit of width.

The atomic computational unit of a neural network: receives inputs, computes a weighted sum, applies an activation, and emits an output.

An artificial neuron (also called a node or unit) is the fundamental element of a neural network. It receives a vector of real-valued inputs, multiplies each by a learned weight, adds a bias, and passes the result through a nonlinear activation function. The concept is a mathematical abstraction loosely analogous to the biological neuron — the analogy is useful for naming but must not be over-read biologically. A single neuron implements a parameterized scalar function: output = activation(w · x + b). Layers of neurons stacked and connected form all feedforward and recurrent architectures.

Techniques that normalize activations within a mini-batch (batch norm) or across features for a single example (layer norm) to stabilize training and accelerate convergence.

Normalization layers re-center and rescale neural network activations to have zero mean and unit variance, then apply learned scale (γ) and shift (β) parameters: normalized = γ * (x - μ) / σ + β. Batch normalization (Ioffe & Szegedy, arXiv:1502.03167) normalizes across the batch dimension for each feature; this reduces internal covariate shift and allows much higher learning rates. Layer normalization (Ba et al., 2016) normalizes across the feature dimension for each example, making it batch-size-independent — the standard in transformers. Normalization is inserted after the linear transformation and before (or after) the activation. Without normalization, deep networks are sensitive to weight initialization and learning rate, often diverging or training very slowly. Normalization is not a standalone cure: a poorly initialized or poorly structured network can still fail even with normalization.

The rule by which gradients are converted into weight updates; variants include SGD, Adam, and RMSProp, each with different momentum and adaptive learning-rate behaviors.

The optimizer step is the rule that converts the gradient ∂L/∂w (computed by backpropagation) into an actual weight update. Plain gradient descent applies w ← w - η·g where g = ∂L/∂w. Practical optimizers add momentum (SGD+momentum: accumulate a velocity term to smooth updates), adaptive learning rates (Adam: maintain per-parameter running estimates of gradient mean and variance; RMSProp: normalize by recent gradient magnitude), or both. Adam (Kingma & Ba, arXiv:1412.6980) is the default for most modern deep learning because it adapts the effective learning rate per parameter, converging faster on problems where gradient magnitudes vary widely across parameters. The optimizer step occurs AFTER backpropagation; it is logically separate from gradient computation.

The failure mode where a high-capacity network memorizes training data rather than learning generalizable patterns, causing high training accuracy but poor test performance.

Overfitting occurs when a neural network's capacity is large relative to the training data size — the network learns to fit the idiosyncratic noise of the training set rather than the underlying data-generating distribution. Symptoms: training loss decreasing while validation loss increases (the loss divergence gap). Neural networks are especially susceptible because their parameter count can far exceed the training example count. Mitigations native to the substrate: dropout (randomly drops units, preventing co-adaptation), L2 weight regularization (penalizes large weights in the loss), data augmentation (increases effective training set size), and early stopping (halt training when validation loss starts rising). Importantly, modern large overparameterized networks (billions of parameters, millions of examples) often show benign overfitting — they interpolate the training data but still generalize — challenging classical bias-variance intuitions.

The high-dimensional space of all possible weight configurations of a network; training is the process of finding a point in this space with low loss and good generalization.

The parameter space (weight space) of a neural network is the Euclidean space ℝᴰ where D = total number of learnable parameters (weights and biases). Each point in this space corresponds to a fully specified network with a specific input-output function. Training by gradient descent traces a path through this space from the initialization point toward regions of lower loss. The parameter space of modern networks is extraordinarily high-dimensional — GPT-3 has D ≈ 175 billion — yet gradient descent reliably finds solutions that generalize. This is theoretically surprising: classical statistical learning theory suggests high-dimensional spaces should be hard to search and prone to overfitting, but the combination of the implicit regularization of SGD, the structure of the loss landscape (few isolated local minima; mostly connected flat valleys), and the geometry of overparameterized models partially explains the empirical success.

The simplest trainable single-neuron binary classifier; the historical origin of the neural network, using a threshold activation and a Hebbian-style update rule.

The perceptron, introduced by Rosenblatt (1958), is a single artificial neuron with a binary threshold activation (step function) trained by the perceptron learning rule: adjust weights when the unit misclassifies an input. It can separate any linearly separable binary classification problem, but fails on non-linearly separable ones (XOR). The perceptron is the conceptual origin of the MLP: stacking multiple perceptron-like units with differentiable activations and training by backpropagation overcomes the linearity limitation. Every modern neural network unit is a generalization of the perceptron with richer activations and gradient-based learning.

The capacity of a neural network to automatically learn useful intermediate representations (features) of the input, without hand-engineering features.

Representation learning is the principle that a neural network's hidden layers learn useful intermediate transformations of the input data automatically from the training signal, without hand-crafted feature engineering. Each hidden layer builds a progressively more abstract representation: in a vision network, early layers detect edges and textures; middle layers detect shapes and object parts; late layers encode object identities. This hierarchy of representations is the key empirical success of deep learning — replacing task-specific feature engineering with end-to-end gradient-based learning. The learned representations can transfer: a network trained on ImageNet learns features useful for other visual tasks (transfer learning). Representation learning is why depth (more layers = more abstraction levels) rather than width is the dominant axis of neural architecture design.

A direct addition of a layer's input to its output (y = F(x) + x), enabling gradient flow through arbitrarily deep networks by creating a shortcut path.

A residual connection (He et al., arXiv:1512.03385) adds the input of a layer directly to the layer's output: y = F(x, W) + x, where F is the learned residual function. The key insight: instead of learning the desired transformation H(x) = F(x) + x directly, the layer learns the residual F(x) = H(x) - x, which is easier when H(x) ≈ x (identity mapping is near the solution). Crucially, the + x creates a gradient highway: during backpropagation, gradients can flow directly through the addition without passing through the nonlinear stack, making very deep networks (50–150+ layers) trainable. Residual connections are now universal in deep networks: every transformer layer wraps its sub-layers with residual additions and normalization, and every deep vision network uses them.

A neural network with feedback connections that allows it to process sequences by maintaining a hidden state across time steps.

A Recurrent Neural Network (RNN) extends the feedforward MLP with temporal feedback: a hidden state h_t is computed from the current input x_t and the previous hidden state h_{t-1}, enabling the network to process variable-length sequences. The recurrence relation is: h_t = activation(W_h · h_{t-1} + W_x · x_t + b). Because the same weight matrix is applied at every time step (weight sharing across time), RNNs can generalize across sequence lengths. However, the chain of multiplicative applications over long sequences causes gradients to either vanish or explode during backpropagation through time (BPTT), limiting practical sequence lengths. LSTMs and GRUs were designed to address this. Transformers have largely replaced RNNs for long-sequence tasks but the recurrent formulation remains important for streaming and state-space models.

Attention where the queries, keys, and values all come from the same sequence, allowing each position to attend to all other positions in that sequence.

Self-attention is the specific form of attention used in transformers where the query, key, and value projections all derive from the same input sequence representation. This allows each position in the sequence to attend to every other position — building a contextual representation that captures global dependencies. In multi-head self-attention, the operation is performed in parallel with H different query/key/value projection matrices, and the H output vectors are concatenated and linearly projected. Self-attention has O(n²) complexity in sequence length n (every pair of positions is compared), which limits raw scalability but delivers superior modeling of long-range dependencies versus the O(n) linear cost of RNNs that compress context into a fixed hidden state.

An encoder-decoder (or decoder-only) architecture that replaces recurrence entirely with multi-head self-attention and position-wise feed-forward layers, enabling parallelism and long-range dependency modeling.

The transformer (Vaswani et al., arXiv:1706.03762) replaces the sequential recurrence of RNNs with attention over all positions in parallel, enabling full parallelism during training. Each transformer layer contains two sub-layers: multi-head self-attention (which computes weighted combinations of all positions' representations) and a position-wise feed-forward network (a small MLP applied independently to each position). Residual connections and layer normalization are applied around each sub-layer. Because there is no recurrence, positional encoding is injected to give the model a sense of sequence order. The transformer architecture has become the substrate for all large-scale language models (GPT, BERT, LLaMA) and increasingly for vision (ViT) and multimodal models. It is this World's architecture anchor: the same unit (neuron), the same weighted-sum computation, organized into attention + residual + normalization layers.

A theorem establishing that a sufficiently wide single-hidden-layer MLP with a nonlinear activation can approximate any continuous function on a compact set to arbitrary precision.

The universal approximation theorem (Hornik, Stinchcombe & White, Neural Networks 1989; Cybenko 1989 for sigmoidal activations) proves that a feedforward network with a single hidden layer containing a sufficient number of units with a nonlinear activation function can approximate any continuous function on a compact subset of ℝⁿ to arbitrary precision. The theorem is an existence result — it says such a network exists but not how to find it or how many neurons are needed. Two practical lessons follow: (1) depth is not required in theory, but it is in practice — a single-hidden-layer approximation may require exponentially many neurons versus a deep network; (2) the theorem justifies the use of neural networks as general-purpose function approximators but gives no guidance on generalization (only approximation). It is the theoretical foundation for the claim that neural networks are a sufficiently expressive hypothesis class.

The failure mode where gradients shrink exponentially as they are propagated back through many layers, making early layers of a deep network train extremely slowly or not at all.

The vanishing gradient problem occurs during backpropagation in deep networks: gradients are computed by successive multiplication of Jacobians (the local gradient of each layer). When these Jacobians have eigenvalues < 1 — which happens naturally with sigmoid/tanh activations saturated near their plateaus — the gradient signal decreases exponentially as it propagates backward through layers. The first layers of a 20-layer sigmoid network may receive gradients 10⁻⁹ times smaller than the last layer, effectively stopping learning. The problem scales with depth: every additional layer multiplies by another small Jacobian. Solutions: ReLU activation (gradient = 1 for positive inputs, does not shrink); residual connections (gradient highway around each layer's Jacobian); LSTMs (gated cell-state with additive updates); batch normalization (keeps activations in the active range of sigmoid/tanh). The vanishing gradient problem is the primary reason deep sigmoid networks were impractical before these mitigations.

A learned scalar parameter on each connection between units; determines how strongly one unit's output influences the next unit's input.

A weight is a real-valued scalar parameter associated with a connection between two neurons. During the forward pass, each weight multiplies the corresponding input; the sum of weighted inputs (plus bias) is the pre-activation value fed to the activation function. Weights are the primary learnable parameters of a neural network — backpropagation computes the gradient of the loss with respect to each weight, and gradient descent updates them to minimize the loss. The matrix of weights connecting one layer to the next is the central object in all feedforward computations and the primary determinant of the function a network implements.

The strategy for setting initial weight values before training; critically determines gradient flow and convergence, especially in deep networks.

Weight initialization sets the starting values of a network's weight matrices before gradient descent begins. Poor initialization — weights too large (exploding activations and gradients) or too small (vanishing activations) — can prevent a deep network from training at all. The critical insight: the variance of a layer's output should match its input, so gradients neither explode nor vanish through depth. Xavier/Glorot initialization (Glorot & Bengio, 2010) scales weights as √(2/(fan_in + fan_out)) for tanh layers. Kaiming/He initialization (He et al., 2015) scales as √(2/fan_in) for ReLU layers, accounting for ReLU's zero-for-negative behavior. All weights are initialized from the same random distribution (Gaussian or uniform) because symmetric initialization — all zeros — means all neurons compute the same gradient and break symmetry only with noise.

The linear combination of inputs and weights plus bias that a neuron computes before applying the activation function: z = w · x + b.

The weighted sum (or pre-activation, often notated z) is the linear combination computed by a neuron: z = Σ(wᵢ · xᵢ) + b = w · x + b. It is the input to the activation function. The weighted sum is entirely linear in both inputs and weights; the nonlinearity comes only from the activation. For an entire layer, the computation is a matrix multiplication: Z = XW + b. This matrix-vector form is why modern neural networks can be computed efficiently on GPU hardware with BLAS/cuBLAS. Without the activation function, stacked weighted sums collapse to a single affine transformation, motivating the need for nonlinear activations.