Lesson overview | Previous part | Next part

Linear Transformations: Part 7: Affine Transformations to Appendix B: Abstract Linear Algebra Perspective

7. Affine Transformations

7.1 Beyond Linearity: Affine Maps

A linear map fixes the origin: $T(\mathbf{0}) = \mathbf{0}$. But many practical transformations in geometry and ML need to shift the origin - they include a translation component.

Definition. A function $f: \mathbb{R}^n \to \mathbb{R}^m$ is an affine transformation if it has the form:

$$f(\mathbf{x}) = A\mathbf{x} + \mathbf{b}$$

where $A \in \mathbb{R}^{m \times n}$ is a matrix (the linear part) and $\mathbf{b} \in \mathbb{R}^m$ is a vector (the translation).

Affine maps are linear maps composed with a translation. They are NOT linear (unless $\mathbf{b} = \mathbf{0}$) - they fail the zero test: $f(\mathbf{0}) = \mathbf{b} \neq \mathbf{0}$ when $\mathbf{b} \neq \mathbf{0}$.

Affine subspaces. The image of $\mathbb{R}^n$ under an affine map is an affine subspace - a translate of a linear subspace. Solutions to $A\mathbf{x} = \mathbf{b}$ (when they exist) form an affine subspace: $\mathbf{x}_0 + \ker(A)$.

Composition of affine maps. If $f(\mathbf{x}) = A\mathbf{x} + \mathbf{b}$ and $g(\mathbf{y}) = C\mathbf{y} + \mathbf{d}$, then:

$$(g \circ f)(\mathbf{x}) = g(A\mathbf{x} + \mathbf{b}) = C(A\mathbf{x} + \mathbf{b}) + \mathbf{d} = CA\mathbf{x} + (C\mathbf{b} + \mathbf{d})$$

The composition is affine: linear part $CA$, translation $C\mathbf{b} + \mathbf{d}$.

7.2 Homogeneous Coordinates

The elegant trick to make affine maps linear is to lift to one higher dimension.

Definition. The homogeneous coordinates of $\mathbf{x} \in \mathbb{R}^n$ are $\tilde{\mathbf{x}} = \begin{pmatrix} \mathbf{x} \\ 1 \end{pmatrix} \in \mathbb{R}^{n+1}$.

The augmented matrix. The affine map $f(\mathbf{x}) = A\mathbf{x} + \mathbf{b}$ becomes:

$$\tilde{f}(\tilde{\mathbf{x}}) = \begin{bmatrix} A & \mathbf{b} \\ \mathbf{0}^\top & 1 \end{bmatrix} \begin{pmatrix} \mathbf{x} \\ 1 \end{pmatrix} = \begin{pmatrix} A\mathbf{x} + \mathbf{b} \\ 1 \end{pmatrix}$$

Now $\tilde{f}$ is a linear map in $\mathbb{R}^{n+1}$. The last coordinate is always 1 for "proper" points.

Key benefit: composition becomes matrix multiplication. Two affine maps $\tilde{M}_1, \tilde{M}_2$ compose as:

$$\tilde{M}_2 \tilde{M}_1 = \begin{bmatrix} A_2 & \mathbf{b}_2 \\ \mathbf{0}^\top & 1 \end{bmatrix} \begin{bmatrix} A_1 & \mathbf{b}_1 \\ \mathbf{0}^\top & 1 \end{bmatrix} = \begin{bmatrix} A_2 A_1 & A_2\mathbf{b}_1 + \mathbf{b}_2 \\ \mathbf{0}^\top & 1 \end{bmatrix}$$

No special cases needed - composition is just matrix multiplication.

Inverse of an affine map:

$$\begin{bmatrix} A & \mathbf{b} \\ \mathbf{0}^\top & 1 \end{bmatrix}^{-1} = \begin{bmatrix} A^{-1} & -A^{-1}\mathbf{b} \\ \mathbf{0}^\top & 1 \end{bmatrix}$$

(Valid when $A$ is invertible.)

In computer graphics: Every rigid body transformation (rotation + translation) is an affine map, and sequences of transformations are composed by matrix multiplication in homogeneous coordinates. This is the foundation of 3D graphics pipelines (OpenGL, Vulkan).

7.3 Neural Network Layers as Affine Maps

A single fully-connected layer computes:

$$\mathbf{z} = W\mathbf{x} + \mathbf{b}, \quad \mathbf{a} = \sigma(\mathbf{z})$$

The pre-activation $\mathbf{z} = W\mathbf{x} + \mathbf{b}$ is an affine map from $\mathbb{R}^{d_{\text{in}}}$ to $\mathbb{R}^{d_{\text{out}}}$. In homogeneous coordinates:

$$\tilde{\mathbf{z}} = \begin{bmatrix} W & \mathbf{b} \\ \mathbf{0}^\top & 1 \end{bmatrix} \tilde{\mathbf{x}}$$

Why bias matters. Without bias ($\mathbf{b} = \mathbf{0}$), each layer is linear and the hyperplanes separating classes must pass through the origin. With bias, the decision boundary can be placed anywhere. This is the geometric reason bias terms dramatically increase expressive power.

Multi-layer without activations. A network $\mathbf{z}_L = W_L(\cdots W_2(W_1\mathbf{x} + \mathbf{b}_1) + \mathbf{b}_2 \cdots) + \mathbf{b}_L$ is still affine:

$$\mathbf{z}_L = W_{\text{eff}}\mathbf{x} + \mathbf{b}_{\text{eff}}$$

Multiple affine layers compose to a single affine layer. Depth without nonlinearity gives no expressive benefit.

BatchNorm as affine rescaling. After normalizing to zero mean and unit variance, BatchNorm applies a learned affine transformation $\gamma \mathbf{z} + \beta$ (elementwise). This is an affine map on the normalized activations, restoring the capacity to represent any desired scale and shift.

Embedding layers. An embedding layer $E \in \mathbb{R}^{|V| \times d}$ maps token indices to vectors. Selecting the $i$-th embedding is equivalent to multiplying by the one-hot vector $\mathbf{e}_i$: $E\mathbf{e}_i = \mathbf{e}_{i,:}$ (the $i$-th row). This is linear in the one-hot representation. The unembedding $W_U \in \mathbb{R}^{d \times |V|}$ maps representations to logits: $\mathbf{l} = W_U \mathbf{h}$, a pure linear map.

---

8. The Jacobian as Linear Approximation

8.1 Linearization of Nonlinear Maps

Every differentiable function is "locally linear" - at any point, it looks like a linear map to first order. This principle underlies calculus, optimization, and all of numerical analysis.

Definition (Total Derivative). A function $f: \mathbb{R}^n \to \mathbb{R}^m$ is differentiable at $\mathbf{x}$ if there exists a linear map $Df_{\mathbf{x}}: \mathbb{R}^n \to \mathbb{R}^m$ such that:

$$\lim_{\lVert\mathbf{h}\rVert \to 0} \frac{\lVert f(\mathbf{x} + \mathbf{h}) - f(\mathbf{x}) - Df_{\mathbf{x}}(\mathbf{h})\rVert}{\lVert\mathbf{h}\rVert} = 0$$

The linear map $Df_{\mathbf{x}}$ is the total derivative or Frechet derivative of $f$ at $\mathbf{x}$. The first-order approximation is:

$$f(\mathbf{x} + \mathbf{h}) \approx f(\mathbf{x}) + Df_{\mathbf{x}}(\mathbf{h})$$

For small $\mathbf{h}$, the function looks like an affine map: constant $f(\mathbf{x})$ plus linear correction $Df_{\mathbf{x}}(\mathbf{h})$.

Geometric meaning. At each point $\mathbf{x}$, the total derivative $Df_{\mathbf{x}}$ is the best linear approximation to $f$ near $\mathbf{x}$. It maps directions (tangent vectors at $\mathbf{x}$) to directions (tangent vectors at $f(\mathbf{x})$). This is the pushforward of vectors.

8.2 The Jacobian Matrix

Definition. The matrix representation of $Df_{\mathbf{x}}$ in standard coordinates is the Jacobian matrix:

$$J_f(\mathbf{x}) = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \cdots & \frac{\partial f_1}{\partial x_n} \\ \vdots & \ddots & \vdots \\ \frac{\partial f_m}{\partial x_1} & \cdots & \frac{\partial f_m}{\partial x_n} \end{bmatrix} \in \mathbb{R}^{m \times n}$$

Entry $(i, j)$: $J_{ij} = \frac{\partial f_i}{\partial x_j}$ - how the $i$-th output changes with the $j$-th input.

Shape mnemonic: output dimension $\times$ input dimension: $(m \times n)$ for $f: \mathbb{R}^n \to \mathbb{R}^m$.

Special cases:

• $f: \mathbb{R}^n \to \mathbb{R}$ (scalar function): Jacobian is the row gradient $\nabla f^\top \in \mathbb{R}^{1 \times n}$.
• $f: \mathbb{R} \to \mathbb{R}^m$ (curve): Jacobian is the column tangent vector $f'(t) \in \mathbb{R}^m$.
• $f: \mathbb{R}^n \to \mathbb{R}^n$ (same dim): $J_f$ is square; $\det(J_f)$ is the local volume scaling.

The Jacobian of softmax. For $\mathbf{s} = \operatorname{softmax}(\mathbf{z})$ where $s_i = e^{z_i} / \sum_k e^{z_k}$:

$$J_{\operatorname{softmax}}(\mathbf{z}) = \operatorname{diag}(\mathbf{s}) - \mathbf{s}\mathbf{s}^\top$$

This is a rank-deficient matrix (rank $n-1$) because softmax outputs sum to 1: the Jacobian has a constant vector $\mathbf{1}$ in its null space.

The Jacobian of ReLU. For $\mathbf{a} = \operatorname{ReLU}(\mathbf{z})$ (elementwise):

$$J_{\operatorname{ReLU}}(\mathbf{z}) = \operatorname{diag}(\mathbf{1}[\mathbf{z} > 0])$$

A diagonal matrix with 1s where $z_i > 0$ and 0s elsewhere. ReLU's Jacobian is a projection - it zeroes out the gradient for dead neurons.

The Jacobian of layer normalization. More complex: layer norm applies mean subtraction, variance normalization, and an affine transformation. Its Jacobian is a projection onto a specific hyperplane (orthogonal to the constant vector), scaled and shifted.

8.3 Chain Rule = Composition of Jacobians

The chain rule for vector-valued functions states: if $f: \mathbb{R}^n \to \mathbb{R}^m$ and $g: \mathbb{R}^m \to \mathbb{R}^p$, then:

$$J_{g \circ f}(\mathbf{x}) = J_g(f(\mathbf{x})) \cdot J_f(\mathbf{x})$$

This is matrix multiplication of Jacobians. The composition of two differentiable maps has a Jacobian equal to the matrix product of their individual Jacobians (evaluated at the appropriate points).

Backpropagation is reverse-mode Jacobian accumulation. For a network $\mathbf{y} = f_L \circ \cdots \circ f_1(\mathbf{x})$ with loss $\ell = L(\mathbf{y})$:

$$\frac{\partial \ell}{\partial \mathbf{x}} = J_{f_1}^\top \cdots J_{f_L}^\top \nabla_{\mathbf{y}} L$$

Reading right to left: start with $\nabla_{\mathbf{y}} L$, multiply by $J_{f_L}^\top$, then $J_{f_{L-1}}^\top$, etc. At each step, we multiply by the transpose Jacobian of a layer - which is the dual map of that layer's linear approximation.

Computational graph view. Each node in the computation graph stores its local Jacobian. Forward pass evaluates the functions; backward pass multiplies the transpose Jacobians (right to left). This is the mathematical content of autograd.

BACKPROPAGATION AS JACOBIAN CHAIN
========================================================================

  Forward:  x --J_1---> z_1 --J_2---> z_2 -- ... --J_L---> y ---> ell

  Backward: \partialell/\partialx <---J_1^T-- \partialell/\partialz_1 <---J_2^T-- \partialell/\partialz_2 <--- ... <---J_L^T-- \nablay L

  Each backward step:  (grad at input) = J^T \times (grad at output)
                     = (dual map) applied to incoming gradient

========================================================================

Why Jacobians matter for training dynamics:

• Large Jacobian singular values -> exploding gradients
• Small Jacobian singular values -> vanishing gradients
• The spectral norm of $J_f$ measures how much the linear approximation of $f$ can amplify inputs

For AI: Gradient clipping, careful weight initialization (Xavier, He), and residual connections all address the Jacobian conditioning problem. Residual connections add an identity Jacobian contribution: $J_{x + F(x)} = I + J_F$, which prevents the singular values from collapsing to zero (vanishing gradients).

---

9. Applications in Machine Learning

9.1 Attention as Linear Projections

The scaled dot-product attention mechanism (Vaswani et al., 2017) is built entirely from linear transformations applied to a sequence of token representations.

Setup. Given an input sequence $X \in \mathbb{R}^{L \times d}$ (L tokens, d-dimensional residual stream), three learned projection matrices $W_Q, W_K \in \mathbb{R}^{d \times d_k}$ and $W_V \in \mathbb{R}^{d \times d_v}$ define:

$$Q = XW_Q, \quad K = XW_K, \quad V = XW_V$$

Each is a linear transformation: $Q$ projects each token into "query space", $K$ into "key space", $V$ into "value space".

Attention scores. The attention pattern $\alpha = \operatorname{softmax}(QK^\top / \sqrt{d_k})$ is a soft selection matrix. For a fixed query vector $\mathbf{q}$, the attention scores $\mathbf{q}^\top K^\top$ are dot products in key space - a linear functional applied to each key.

Output. $\operatorname{Attn}(Q,K,V) = \alpha V$. The output is a weighted linear combination of value vectors - a linear operation parameterized by $\alpha$. The output projection $W_O \in \mathbb{R}^{d_v \times d}$ maps back to the residual stream: another linear map.

Multi-head attention. Each head $h$ applies its own projection matrices $(W_Q^h, W_K^h, W_V^h)$, computes attention independently, and the results are concatenated and projected:

$$\operatorname{MHA}(X) = \operatorname{Concat}(\operatorname{head}_1, \ldots, \operatorname{head}_H) W_O$$

This is a composition and concatenation of linear maps. The expressivity comes from the multiplicative interaction $QK^\top$ (which is bilinear, not linear) - but conditioned on fixed $\alpha$, the rest is linear.

For AI: The "OV circuit" and "QK circuit" decomposition (Elhage et al., 2021) analyzes transformer attention by studying the linear maps $W_OW_V$ (value writing) and $W_QW_K^\top$ (key-query matching) separately. This is possible precisely because attention is compositionally linear.

9.2 LoRA: Fine-Tuning via Low-Rank Composition

Low-Rank Adaptation (Hu et al., 2021) is one of the most important parameter-efficient fine-tuning methods, and its design is a direct application of linear map theory.

Setup. Freeze the pretrained weight $W_0 \in \mathbb{R}^{d \times k}$. Add a trainable low-rank update:

$$W = W_0 + \Delta W = W_0 + BA$$

where $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$, and $r \ll \min(d, k)$.

Rank-nullity interpretation. The map $\Delta W = BA$ has rank at most $r$. By rank-nullity:

$$\dim(\ker(\Delta W)) \geq k - r$$

The update only changes the network's behavior along at most $r$ directions in the $k$-dimensional input space. When $r = 8$ and $k = 4096$, only $8/4096 \approx 0.2\%$ of input directions are affected - the update is extremely sparse in "direction space."

The two-layer interpretation. The update $\Delta W = BA$ is a composition of two maps:

1. $A: \mathbb{R}^k \to \mathbb{R}^r$ - compression to rank-$r$ space
2. $B: \mathbb{R}^r \to \mathbb{R}^d$ - expansion back to full dimension

Initializing $A$ randomly and $B = 0$ ensures $\Delta W = 0$ at the start of fine-tuning, so the model starts from the pretrained weights.

Parameter count. $\Delta W$ would have $dk$ parameters. $BA$ uses $dr + rk = r(d+k)$ parameters. Savings ratio: $\frac{r(d+k)}{dk} = \frac{r}{d} + \frac{r}{k} \approx \frac{2r}{\min(d,k)}$. For $r=8$, $d=k=4096$: savings factor of $\approx 256\times$.

Extensions: DoRA (Weight-Decomposition Low-Rank Adaptation, 2024) decomposes $W$ into magnitude and direction, applying LoRA only to the direction component. GaLore (2024) applies LoRA-style updates to gradients rather than weights.

9.3 Linear Probes and the Linear Representation Hypothesis

Linear probing tests whether a feature is linearly decodable from a model's representations. Given representations $\{\mathbf{h}_i\}$ and labels $\{y_i\}$, train a linear classifier:

$$\hat{y} = \operatorname{sign}(\mathbf{w}^\top \mathbf{h} + b)$$

If the probe achieves high accuracy, the feature is linearly represented - it corresponds to a direction $\mathbf{w}$ in representation space.

The Linear Representation Hypothesis (Mikolov et al., 2013; Elhage et al., 2022; Park et al., 2023) states that high-level features (sentiment, syntax, factual attributes, world models) are encoded as directions in the residual stream - i.e., as linear features.

Evidence:

• Word2vec arithmetic: $\mathbf{v}(\text{king}) - \mathbf{v}(\text{man}) + \mathbf{v}(\text{woman}) \approx \mathbf{v}(\text{queen})$. Semantic relationships are linear offsets.
• Steering vectors: adding $c\mathbf{d}$ to all residual stream activations controls model behavior (e.g., "banana" direction, sentiment directions).
• Probing studies: most tested syntactic and semantic features are linearly decodable.

Superposition. When there are more features than dimensions, the model stores features in near-orthogonal directions that partially overlap (superposition). This is still linear representation - just with interference.

For AI: If the linear representation hypothesis holds broadly, then:

• Linear algebra provides the right toolkit for model interpretability.
• Interventions on model behavior reduce to vector addition in representation space.
• Feature extraction is a linear map - PCA/SVD on activations finds meaningful directions.

9.4 Embedding and Unembedding

Token embeddings. The embedding matrix $W_E \in \mathbb{R}^{|V| \times d}$ maps vocabulary indices to $d$-dimensional vectors. Indexing row $i$ of $W_E$ is equivalent to the linear map $\mathbf{e}_i^\top W_E$ (one-hot selection). The embedding layer is linear in the one-hot representation.

Unembedding. The unembedding matrix $W_U \in \mathbb{R}^{d \times |V|}$ maps residual stream vectors to logits over the vocabulary:

$$\mathbf{l} = W_U \mathbf{h} \in \mathbb{R}^{|V|}$$

This is a pure linear map. The logit for token $v$ is $l_v = \mathbf{w}_{U,v}^\top \mathbf{h}$ - a dot product (linear functional) between the unembedding direction $\mathbf{w}_{U,v}$ and the residual stream.

Logit lens. Applying $W_U$ to intermediate residual stream states (before the final layer) gives "early predictions" - showing what the model is computing at each layer. This technique (Nostalgebraist, 2020) is possible because unembedding is linear.

Tied embeddings. Many models (GPT-2, LLaMA variants) use $W_U = W_E^\top$ - the same matrix for both embedding and unembedding. This enforces consistency: the most likely next token $v$ after seeing context $\mathbf{h}$ is the one whose embedding $W_E[v]$ has the highest dot product with $\mathbf{h}$ - i.e., $\operatorname{argmax}_v W_E[v] \cdot \mathbf{h}$.

---

10. Common Mistakes

---

11. Exercises

Exercise 1 * - Kernel and Image Computation

Goal: Given explicit matrices, compute kernel and image and verify rank-nullity.

(a) For $A = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{bmatrix}$: find a basis for $\ker(A)$, a basis for $\operatorname{im}(A)$, and verify rank + nullity = 3.

(b) For $B = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix}$ (projection to first two coordinates): identify kernel and image without computation.

(c) For the differentiation map $D: \mathcal{P}_3 \to \mathcal{P}_2$ with matrix as in 3.1: find kernel and image dimensions.

Exercise 2 * - Matrix of a Linear Map in Non-Standard Basis

Goal: Construct the matrix of a given transformation in a specified basis.

Let $T: \mathbb{R}^2 \to \mathbb{R}^2$ be reflection across the line $y = x$ (i.e., $T(x,y) = (y,x)$). Basis $\mathcal{B} = \{(1,1)/\sqrt{2}, (1,-1)/\sqrt{2}\}$.

(a) Write the standard matrix $A$ of $T$ in the standard basis.

(b) Compute the change-of-basis matrix $P$ from $\mathcal{B}$ to the standard basis.

(c) Compute $[T]_{\mathcal{B}} = P^{-1}AP$. What is notable about this result?

(d) Explain geometrically why $T$ is diagonal in the basis $\mathcal{B}$.

Exercise 3 * - Rank-Nullity and Linear Systems

Goal: Use rank-nullity to understand solution sets of linear systems.

(a) $A \in \mathbb{R}^{4 \times 6}$ has rank 3. What is the nullity? How many free variables are there in $A\mathbf{x} = \mathbf{0}$?

(b) $B \in \mathbb{R}^{6 \times 4}$ has rank 4. Is $B\mathbf{x} = \mathbf{b}$ always solvable? What is the nullity of $B$?

(c) Prove: if $T: V \to V$ is a linear map on a finite-dimensional space, then $T$ is injective if and only if $T$ is surjective.

Exercise 4 ** - Projection Operator Construction

Goal: Build an orthogonal projection onto a given subspace and verify idempotency.

Let $S = \operatorname{span}\{(1, 2, 0), (0, 1, 1)\} \subseteq \mathbb{R}^3$.

(a) Orthonormalize the spanning set using Gram-Schmidt.

(b) Construct the projection matrix $P = QQ^\top$ where $Q$ has the orthonormal basis as columns.

(c) Verify: $P^2 = P$, $P = P^\top$, and $\operatorname{rank}(P) = \operatorname{tr}(P) = 2$.

(d) Compute $(I-P)$ and verify it is also a projection. What does it project onto?

Exercise 5 ** - Jacobian of Softmax

Goal: Derive and verify the Jacobian of the softmax function.

(a) Derive $\frac{\partial s_i}{\partial z_j}$ for $\mathbf{s} = \operatorname{softmax}(\mathbf{z})$, handling the cases $i = j$ and $i \neq j$ separately.

(b) Show that the Jacobian is $J = \operatorname{diag}(\mathbf{s}) - \mathbf{s}\mathbf{s}^\top$.

(c) Verify that $J\mathbf{1} = \mathbf{0}$ (the Jacobian kills constant vectors). Why does this make sense?

(d) Show that the rank of $J$ is at most $n-1$.

Exercise 6 ** - Affine Map Composition via Homogeneous Coordinates

Goal: Compose affine transformations using augmented matrices.

In $\mathbb{R}^2$, let:

• $f_1$: rotation by $45 degrees$ then translate by $(1, 0)$
• $f_2$: scale by $2$ in both dimensions then translate by $(0, 1)$

(a) Write augmented $3 \times 3$ matrices $\tilde{M}_1$ and $\tilde{M}_2$ for $f_1$ and $f_2$.

(b) Compute $\tilde{M}_2 \tilde{M}_1$ (apply $f_1$ then $f_2$). What are the effective rotation, scale, and translation?

(c) Apply the composition to the point $(1, 0)$. Verify by applying $f_1$ then $f_2$ directly.

Exercise 7 *** - LoRA Rank Analysis

Goal: Analyze the geometry of low-rank weight updates.

Let $W_0 \in \mathbb{R}^{512 \times 512}$ be a pretrained weight. A LoRA update with rank $r = 4$ is $\Delta W = BA$ where $B \in \mathbb{R}^{512 \times 4}$, $A \in \mathbb{R}^{4 \times 512}$.

(a) What is $\operatorname{rank}(\Delta W)$? What is the nullity of $\Delta W$?

(b) Generate random $B$ and $A$ and numerically verify: $\operatorname{rank}(BA) \leq 4$.

(c) For a fixed input $\mathbf{x} \in \mathbb{R}^{512}$, compute $\Delta W \mathbf{x} = BA\mathbf{x}$. Show this is in $\operatorname{col}(B)$.

(d) Compute the singular values of $\Delta W$. How many are nonzero? What does this tell you about the effective dimension of the update?

(e) Compare the number of trainable parameters: $512 \times 512$ vs $r(512 + 512)$ for $r = 4, 8, 16, 32$.

Exercise 8 *** - Linear Probing and the Linear Representation Hypothesis

Goal: Empirically test whether a concept is linearly represented in embedding space.

(a) Generate synthetic embeddings for binary sentiment: positive embeddings clustered near $\mathbf{d}$, negative near $-\mathbf{d}$ for a fixed direction $\mathbf{d}$, plus noise.

(b) Train a linear probe (logistic regression on top of embeddings) and measure accuracy.

(c) Apply PCA to the embeddings. Show that the first principal component aligns with $\mathbf{d}$.

(d) Add a "superposition" condition: embed two independent binary features in $d = 3$ dimensions. Show that both features can be linearly decoded but with interference.

(e) Compute the mutual coherence of the feature directions. How does it relate to probe accuracy?

---

12. Why This Matters for AI (2026 Perspective)

---

13. Conceptual Bridge

Looking Back

This section builds on two foundational pillars from earlier in the curriculum.

From Chapter 2 06: Vector Spaces and Subspaces: We studied the axioms of abstract vector spaces (closure, associativity, identity, inverse, distributivity) and their subspaces (spans, null spaces, column spaces). Linear transformations are the maps between these abstract structures - the morphisms of the category of vector spaces. The four fundamental subspaces (column space, null space, row space, left null space) that were defined for matrices are now understood as $\operatorname{im}(T)$, $\ker(T)$, $\operatorname{im}(T^\top)$, and $\ker(T^\top)$ - intrinsic properties of the map, not the matrix.

From 01: Eigenvalues and Eigenvectors and 02: SVD: Both eigendecomposition and SVD are studied as decompositions of linear maps into simple pieces. Eigendecomposition finds a basis in which $T$ acts by scaling. SVD finds two orthonormal bases (for domain and codomain) in which $T$ acts by scaling. These are special cases of the general change-of-basis machinery developed here in 3.

From 03: PCA: PCA uses the linear structure of the data covariance matrix - the covariance $C = X^\top X / (n-1)$ is built from linear maps - to find the principal directions via SVD. The whitening transform and PCA projection are linear maps, and their geometry (dimension reduction, preserved variance) follows directly from the rank-nullity theorem.

Looking Forward

The abstract machinery of linear transformations will appear throughout the rest of the curriculum in concrete technical forms.

05: Orthogonality and Orthonormality: Gram-Schmidt is an algorithm that constructs a specific linear map (change of basis to an orthonormal basis). QR decomposition $A = QR$ factors a linear map as an orthogonal map $Q$ followed by a triangular map $R$. Orthogonal projections (5.1 here) are studied in depth there.

06: Matrix Norms: The spectral norm $\lVert A \rVert_2 = \sigma_1(A)$ measures how much a linear map can amplify vectors. The nuclear norm $\lVert A \rVert_* = \sum \sigma_i$ measures the "effective rank." These norms quantify properties of the linear map that are invisible from the matrix entries.

Chapter 4: Calculus Fundamentals: The Jacobian (8 here) is the bridge between linear algebra and calculus. Multivariable calculus is essentially the study of how linear maps (Jacobians) approximate smooth nonlinear maps. The Hessian is the second-order analogue - a bilinear map that measures curvature. The implicit function theorem, inverse function theorem, and change-of-variables formula all rely on the Jacobian being an invertible linear map at the point of interest.

Curriculum Position

POSITION IN THE MATH FOR LLMS CURRICULUM
========================================================================

  Chapter 1: Mathematical Foundations
  Chapter 2: Linear Algebra Basics
    +-- Vectors, Matrix Operations, Systems of Equations
    +-- Determinants, Matrix Rank
    +-- Vector Spaces ----------------------------+
                                                  | prerequisite
  Chapter 3: Advanced Linear Algebra              |
    +-- 01-Eigenvalues --------------------------+
    +-- 02-SVD ---------------------------------+
    +-- 03-PCA ---------------------------------+
    |                                             |
    +-- 04-Linear Transformations <--------------+
    |      (YOU ARE HERE)
    |      Kernel, Image, Rank-Nullity
    |      Matrix Representation, Change of Basis
    |      Composition, Isomorphisms, Jacobian
    |      Affine Maps, Dual Spaces, AI Applications
    |                                             |
    +-- 05-Orthogonality <----------------------+
    +-- 06-Matrix Norms <-----------------------+
    +-- 07-Positive Definite Matrices           |
    +-- 08-Matrix Decompositions               |
                                               down
  Chapter 4: Calculus Fundamentals
    (Jacobians and chain rule developed further)

========================================================================

The unifying theme: Every major algorithm in deep learning is a composition of linear maps and nonlinearities. Understanding this section means understanding the language in which every neural network, every attention mechanism, every gradient computation, and every interpretability method is written. The matrix is not the territory - but knowing how to move between coordinate representations (change of basis), how to measure what a map collapses (kernel and rank), how to compose maps (matrix multiplication), and how to invert them (isomorphisms and pseudo-inverses) gives you the full algebraic toolkit to reason about any linear system you encounter in machine learning.

---

<- PCA | Back to Advanced Linear Algebra | Orthogonality ->

---

Appendix A: Extended Examples and Computations

A.1 Computing the Matrix of Differentiation

We work out the differentiation operator in full detail to build intuition for linear maps on function spaces.

Setting. Let $V = \mathcal{P}_3 = \{a_0 + a_1 x + a_2 x^2 + a_3 x^3\}$ with basis $\mathcal{B}_V = \{1, x, x^2, x^3\}$ and $W = \mathcal{P}_2$ with basis $\mathcal{B}_W = \{1, x, x^2\}$.

Define $D: V \to W$ by $D(p) = p'$ (differentiation).

Step 1: Apply $D$ to each basis vector of $V$.

• $D(1) = 0 = 0 \cdot 1 + 0 \cdot x + 0 \cdot x^2$ -> coordinate vector $(0, 0, 0)^\top$
• $D(x) = 1 = 1 \cdot 1 + 0 \cdot x + 0 \cdot x^2$ -> coordinate vector $(1, 0, 0)^\top$
• $D(x^2) = 2x = 0 \cdot 1 + 2 \cdot x + 0 \cdot x^2$ -> coordinate vector $(0, 2, 0)^\top$
• $D(x^3) = 3x^2 = 0 \cdot 1 + 0 \cdot x + 3 \cdot x^2$ -> coordinate vector $(0, 0, 3)^\top$

Step 2: Assemble into matrix.

$$[D]_{\mathcal{B}_V}^{\mathcal{B}_W} = \begin{bmatrix} 0 & 1 & 0 & 0 \\ 0 & 0 & 2 & 0 \\ 0 & 0 & 0 & 3 \end{bmatrix}$$

Step 3: Use the matrix. Differentiate $p(x) = 2 + 3x - x^2 + 4x^3$. In $\mathcal{B}_V$-coordinates: $[p]_{\mathcal{B}_V} = (2, 3, -1, 4)^\top$.

$$[D(p)]_{\mathcal{B}_W} = \begin{bmatrix} 0 & 1 & 0 & 0 \\ 0 & 0 & 2 & 0 \\ 0 & 0 & 0 & 3 \end{bmatrix} \begin{pmatrix} 2 \\ 3 \\ -1 \\ 4 \end{pmatrix} = \begin{pmatrix} 3 \\ -2 \\ 12 \end{pmatrix}$$

So $D(p) = 3 - 2x + 12x^2$. Check: $(2 + 3x - x^2 + 4x^3)' = 3 - 2x + 12x^2$. OK

Kernel of $D$: The null space is $\{p : p' = 0\} = \{\text{constants}\} = \operatorname{span}\{1\}$. Dimension 1.

Image of $D$: All polynomials of degree $\leq 2$ (since every $q \in \mathcal{P}_2$ equals $(p)'$ for some $p \in \mathcal{P}_3$). Dimension 3.

Rank-nullity check: $\dim(\mathcal{P}_3) = 4 = 1 + 3 = \text{nullity} + \text{rank}$. OK

A.2 Change of Basis: Full Worked Example

Problem. Let $T: \mathbb{R}^2 \to \mathbb{R}^2$ rotate vectors by $30 degrees$ counterclockwise. Express $T$ in the rotated basis $\mathcal{B}' = \{R_{45 degrees }\mathbf{e}_1, R_{45 degrees }\mathbf{e}_2\}$.

Step 1: Standard matrix of $T$.

$$A = R_{30 degrees } = \begin{bmatrix} \cos 30 degrees & -\sin 30 degrees \\ \sin 30 degrees & \cos 30 degrees \end{bmatrix} = \begin{bmatrix} \sqrt{3}/2 & -1/2 \\ 1/2 & \sqrt{3}/2 \end{bmatrix}$$

Step 2: Change-of-basis matrix. The new basis vectors, in standard coordinates:

$$\mathbf{b}'_1 = R_{45 degrees }\mathbf{e}_1 = (\cos 45 degrees , \sin 45 degrees )^\top = (1/\sqrt{2}, 1/\sqrt{2})^\top$$ $$\mathbf{b}'_2 = R_{45 degrees }\mathbf{e}_2 = (-\sin 45 degrees , \cos 45 degrees )^\top = (-1/\sqrt{2}, 1/\sqrt{2})^\top$$ $$P = \begin{bmatrix} 1/\sqrt{2} & -1/\sqrt{2} \\ 1/\sqrt{2} & 1/\sqrt{2} \end{bmatrix} = R_{45 degrees }$$

Step 3: New matrix.

$$[T]_{\mathcal{B}'} = P^{-1} A P = R_{-45 degrees } R_{30 degrees } R_{45 degrees }$$

Since rotation matrices commute (they all rotate around the same axis in 2D): $R_{-45 degrees } R_{30 degrees } R_{45 degrees } = R_{30 degrees }$.

Key insight: A rotation in 2D has the same matrix in every orthonormal basis (since all such matrices are just $R_\theta$). This is because $R_\theta$ commutes with all rotations: $R_\alpha R_\theta R_\alpha^{-1} = R_\theta$.

A.3 The Null Space as a Subspace: Visualization

For $A = \begin{bmatrix} 1 & 2 & -1 \\ 2 & 4 & -2 \end{bmatrix}$ (rows are multiples), let's find $\ker(A)$.

Row reduce: $R_2 \leftarrow R_2 - 2R_1$:

$$\begin{bmatrix} 1 & 2 & -1 \\ 0 & 0 & 0 \end{bmatrix}$$

Free variables: $x_2 = s$, $x_3 = t$ (free). Back-substitute: $x_1 = -2s + t$.

$$\ker(A) = \left\{ s \begin{pmatrix} -2 \\ 1 \\ 0 \end{pmatrix} + t \begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix} : s, t \in \mathbb{R} \right\}$$

This is a plane through the origin in $\mathbb{R}^3$. The map $T_A: \mathbb{R}^3 \to \mathbb{R}^2$ collapses this entire plane to zero. By rank-nullity: rank $= 1$, nullity $= 2$, and $1 + 2 = 3 = \dim(\mathbb{R}^3)$. OK

The image of $T_A$ is $\operatorname{span}\{(1,2)^\top\}$ - a line in $\mathbb{R}^2$ - since the two columns of $A$ are $(1,2)^\top$ and $(2,4)^\top = 2 \cdot (1,2)^\top$, so rank = 1.

---

Appendix B: Abstract Linear Algebra Perspective

B.1 The Category of Vector Spaces

In the language of category theory (which provides a unifying framework for all of mathematics):

• Objects: Vector spaces over a field $\mathbb{F}$
• Morphisms: Linear maps between them
• Composition: Function composition (= matrix multiplication)
• Identity morphism: The identity map $\operatorname{id}_V(\mathbf{v}) = \mathbf{v}$ (= identity matrix $I$)

This forms a category denoted $\mathbf{Vect}_\mathbb{F}$.

Isomorphisms in the category are exactly the invertible linear maps - this matches our definition of isomorphism in 4.3.

Functors are maps between categories that preserve the categorical structure. The "matrix representation" is a functor from $\mathbf{Vect}_\mathbb{F}$ (with chosen bases) to the category $\mathbf{Mat}_\mathbb{F}$ of matrices. Changing bases corresponds to a natural transformation.

This perspective matters for ML because: neural network architectures are themselves categorical structures (composition of morphisms), and categorical thinking helps reason about when two architectures are "equivalent" (related by isomorphisms).

B.2 Infinite-Dimensional Extensions

The theory extends to infinite-dimensional spaces, where the familiar finite-dimensional results require modification.

Bounded linear operators. On Hilbert spaces (infinite-dimensional inner product spaces), the right notion of "linear transformation" is a bounded linear operator $T: H \to H$ satisfying $\lVert T\mathbf{v}\rVert \leq M\lVert\mathbf{v}\rVert$ for some $M < \infty$. Unbounded operators (like differentiation on $L^2$) require careful domain specification.

The spectral theorem for compact operators. A compact self-adjoint operator on a Hilbert space has a countable set of eigenvalues $\lambda_1 \geq \lambda_2 \geq \cdots \to 0$ and an orthonormal basis of eigenvectors. This is the infinite-dimensional analogue of diagonalization.

Functional analysis. The study of linear maps on infinite-dimensional spaces is called functional analysis. Key results (Hahn-Banach, open mapping theorem, closed graph theorem) parallel finite-dimensional results but require additional technical hypotheses.

For AI: Neural network function classes are subsets of infinite-dimensional function spaces ($L^2$ or Sobolev spaces). Understanding the "size" and "complexity" of these classes uses infinite-dimensional linear algebra - e.g., kernel methods and Gaussian processes operate in reproducing kernel Hilbert spaces (RKHS), and neural tangent kernel theory analyzes infinitely wide networks using spectral theory of linear operators.

B.3 The Tensor Product and Multilinear Maps

Bilinear maps. A map $B: V \times W \to U$ is bilinear if it is linear in each argument separately. The attention score $B(\mathbf{q}, \mathbf{k}) = \mathbf{q}^\top \mathbf{k}$ is bilinear (linear in $\mathbf{q}$, linear in $\mathbf{k}$, but not linear jointly: $B(c\mathbf{q}, c\mathbf{k}) = c^2 B(\mathbf{q}, \mathbf{k}) \neq c B(\mathbf{q}, \mathbf{k})$ in general).

The tensor product $V \otimes W$ is the universal space for bilinear maps: every bilinear map $B: V \times W \to U$ factors through a unique linear map $\tilde{B}: V \otimes W \to U$. This is why tensors (in the ML sense: multi-dimensional arrays) are called tensors - they represent multilinear maps.

For AI: The key operation in self-attention, $\mathbf{q}^\top \mathbf{k} = \langle \mathbf{q}, \mathbf{k} \rangle$, is a bilinear form. The matrix $W$ in the "weight matrix" formulation of attention ($\mathbf{q}^\top W_{\text{QK}} \mathbf{k}$) makes this explicit: $W_{\text{QK}} = W_Q^\top W_K$ is the matrix of the bilinear form. This is the reason attention is more expressive than standard linear transformations - it computes a bilinear (quadratic) function of the input.

---

---