Mathematics for AI/ML/LLM — Cheatsheet

Every formula that actually matters. No filler. Organized by how you encounter them in practice.

1 · Linear Algebra

Vectors

Matrices

Decompositions

Eigendecomposition — Used in: PCA, spectral clustering, graph analysis

SVD — Used in: PCA, LoRA, matrix compression, recommender systems

• $U$ ($m \times m$): left singular vectors
• $\Sigma$ ($m \times n$): singular values on diagonal
• $V^T$ ($n \times n$): right singular vectors
• Low-rank approximation: $A_k = U_k \Sigma_k V_k^T$ (keep top-$k$ values)

PCA via SVD: Center data $X$, compute SVD, project onto top-$k$ columns of $V$.

Matrix Properties That Matter

2 · Calculus

Derivatives You Need to Know

Rules That Drive Backpropagation

The chain rule IS backpropagation. Every layer computes local gradient × upstream gradient.

Multivariate Calculus

Gradient — direction of steepest ascent, used in every optimizer:

Jacobian ($\mathbf{f}: \mathbb{R}^n \to \mathbb{R}^m$) — used in normalizing flows, diffusion models:

Hessian — second-order info, used in loss landscape analysis, Newton's method:

• $H \succ 0$ (positive definite) → local minimum
• $H \prec 0$ (negative definite) → local maximum
• Mixed eigenvalues → saddle point (common in deep learning!)

3 · Probability & Statistics

Core Rules

Bayes is everywhere: Naive Bayes, MAP estimation, Bayesian neural nets, posterior inference.

Key Distributions

Estimation

MLE — maximize likelihood of observed data:

MAP — MLE + prior belief (= regularization!):

Gaussian prior on $\theta$ → L2 regularization. Laplace prior → L1 regularization.

Expectation & Variance

Covariance

4 · Activation Functions

5 · Loss Functions

Classification

Cross-Entropy (multi-class) — THE standard classification loss:

Binary Cross-Entropy:

Focal Loss — handles class imbalance (used in object detection):

Regression

MSE:

MAE / L1 Loss:

Huber Loss — smooth transition between MSE and MAE:

Contrastive & Embedding Losses

Contrastive Loss (SimCLR / CLIP):

Where $\text{sim}$ = cosine similarity, $\tau$ = temperature

Triplet Loss:

• $a$ = anchor, $p$ = positive, $n$ = negative, $\alpha$ = margin

6 · Optimization

Gradient Descent Variants

Vanilla GD:

SGD with Momentum:

Adam (the default optimizer for most deep learning):

Defaults: $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$

AdamW (Adam with decoupled weight decay — used in LLM training):

Key difference from Adam: weight decay $\lambda\theta_t$ is applied directly, not through the gradient.

Gradient Clipping

By norm (prevents exploding gradients — standard in LLM training):

Learning Rate Schedules

Cosine Annealing (used in most modern LLM training):

Warmup + Cosine Decay (the LLM standard):

Regularization

Convexity

• Convex → single global minimum (guaranteed convergence)
• Deep learning losses are non-convex → saddle points, local minima

7 · Normalization

Batch Normalization

Normalizes across the batch dimension. Used in CNNs.

Layer Normalization

Normalizes across the feature dimension. Used in original Transformers, BERT.

RMSNorm (Root Mean Square Normalization)

No mean subtraction, no bias $\beta$. Faster than LayerNorm. Used in LLaMA, GPT-4, modern LLMs.

8 · Transformer & Attention Math

Scaled Dot-Product Attention

• $Q = XW_Q$ (queries), $K = XW_K$ (keys), $V = XW_V$ (values)
• $d_k$ = key dimension. Scaling by $\sqrt{d_k}$ prevents softmax saturation.
• Complexity: $O(n^2 d)$ where $n$ = sequence length

Why $\sqrt{d_k}$?

If $q, k$ have components with variance 1, then $q \cdot k$ has variance $d_k$. Dividing by $\sqrt{d_k}$ restores unit variance → softmax gets reasonable gradients.

Multi-Head Attention

• $h$ heads, each with $d_k = d_{model}/h$
• Lets the model attend to different representation subspaces

Causal (Autoregressive) Masking

Applied before softmax: $\text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + \text{mask}\right)V$

Prevents token $i$ from attending to future tokens $j > i$. Used in GPT, LLaMA, all decoder models.

Transformer Block

Input
  → RMSNorm / LayerNorm
  → Multi-Head Self-Attention + Residual
  → RMSNorm / LayerNorm
  → Feed-Forward Network + Residual
Output

Feed-Forward Network (FFN):

• $W_1$: $d_{model} \to d_{ff}$ (typically $d_{ff} = 4 \times d_{model}$)
• $W_2$: $d_{ff} \to d_{model}$

SwiGLU FFN (used in LLaMA, PaLM, modern LLMs):

Residual Connection

Enables gradient flow through deep networks (50+ layers in LLMs).

9 · LLM-Specific Math

Positional Encoding — Sinusoidal (Original Transformer)

Rotary Position Embedding (RoPE) — Used in LLaMA, GPT-NeoX

Rotates query and key vectors based on position:

Key property: $\langle f(q, m), f(k, n) \rangle$ depends only on $q$, $k$, and relative position $m - n$.

Tokenization (BPE)

Byte-Pair Encoding greedily merges the most frequent adjacent token pair:

Repeat until vocabulary size reached. Subword tokenization balances vocabulary size vs sequence length.

Language Model Probability

Training objective (minimize negative log-likelihood):

import torch
import torch.nn as nn
import torch.nn.functional as F

# === Tensors ===
x = torch.randn(batch, seq_len, d_model)       # Random tensor
x.shape, x.dtype, x.device                      # Inspect

# === Linear Algebra ===
A @ B                                            # Matrix multiply
torch.linalg.svd(A)                              # SVD
torch.linalg.eig(A)                              # Eigendecomposition
torch.linalg.norm(x, dim=-1)                     # Norm

# === Autograd ===
x = torch.tensor([2.0], requires_grad=True)
y = x**2 + 3*x
y.backward()
x.grad                                           # dy/dx = 2x + 3 = 7.0

# === Key Layers ===
nn.Linear(d_in, d_out)                           # Fully connected
nn.Embedding(vocab_size, d_model)                # Embedding lookup
nn.MultiheadAttention(d_model, num_heads)        # Multi-head attention
nn.LayerNorm(d_model)                            # Layer normalization
nn.Dropout(p=0.1)                                # Dropout

# === Loss Functions ===
nn.CrossEntropyLoss()                            # Classification
nn.MSELoss()                                     # Regression
nn.BCEWithLogitsLoss()                           # Binary classification
F.cosine_similarity(a, b)                        # Similarity

# === Optimizers ===
torch.optim.Adam(model.parameters(), lr=1e-4)
torch.optim.AdamW(model.parameters(), lr=1e-4, weight_decay=0.01)

# === LR Schedulers ===
torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=1000)
torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=1e-3, total_steps=10000)