All Courses
11 · GRAPH THEORY

Spectral Graph Theory

"To understand a graph, listen to its spectrum. The eigenvalues of the Laplacian are the resonant frequencies of the graph - they reveal clusters, bottlenecks, expansion, and the rate at which information diffuses across every edge."

Overview

Spectral graph theory is the study of graphs through the eigenvalues and eigenvectors of matrices naturally associated with them - principally the adjacency matrix $A$, the degree matrix $D$, and the graph Laplacian $L = D - A$. The central insight is that algebraic properties of these matrices correspond precisely to combinatorial and geometric properties of the graph: the number of connected components equals the multiplicity of eigenvalue zero; the second-smallest eigenvalue $\lambda_2$ quantifies how "hard" the graph is to disconnect; the eigenvectors of $L$ form a natural Fourier basis for signals defined on the graph.

This connection between spectral algebra and graph topology has made spectral graph theory one of the most productive areas of modern discrete mathematics - and, increasingly, one of the most important mathematical foundations for machine learning. Spectral clustering (Shi & Malik, 2000; Ng, Jordan & Weiss, 2002) remains a gold-standard unsupervised learning method for non-convex clusters. Graph Convolutional Networks (Kipf & Welling, 2017) are derived from first principles as first-order Chebyshev approximations to spectral filters. Laplacian positional encodings power modern graph Transformers (Dwivedi et al., 2022; GPS, 2022). Even language model attention matrices can be analyzed as weighted graphs whose spectral properties reveal information flow.

This section develops the full theory from scratch. We begin with the three fundamental graph matrices and their spectral properties, build up to the Cheeger inequality and expander graphs, construct the graph Fourier transform, derive spectral clustering rigorously, and connect everything to modern AI applications. Students who complete this section will have the mathematical fluency to read GNN papers, design graph-based ML systems, and understand why spectral methods work when they work - and why they fail when they do.

Prerequisites

Learning Objectives

After completing this section, you will:

  • Construct the adjacency matrix $A$, degree matrix $D$, and graph Laplacian $L = D - A$ for any graph, and derive the normalized variants $L_{\text{sym}}$ and $L_{\text{rw}}$
  • Prove that the graph Laplacian is positive semidefinite using the quadratic form $\mathbf{x}^\top L \mathbf{x} = \sum_{(i,j)\in E}(x_i - x_j)^2$
  • State and prove that the multiplicity of eigenvalue $0$ of $L$ equals the number of connected components
  • Define algebraic connectivity $\lambda_2(L)$ and interpret the Fiedler vector as a graph bisection tool
  • State Cheeger's inequality $h^2/2 \leq \lambda_2 \leq 2h$ and explain its implications for expander graphs and random walk mixing
  • Define the Graph Fourier Transform and interpret graph signals in the spectral domain
  • Derive spectral clustering (RatioCut and NCut) from graph partitioning relaxations
  • Implement the Laplacian eigenmaps algorithm and connect it to spectral positional encodings in graph Transformers
  • Derive the GCN layer as a first-order Chebyshev approximation to a spectral filter
  • Analyze PageRank as a spectral problem on directed graphs

Table of Contents

1. Intuition

1.1 Hearing the Shape of a Graph

In 1966, mathematician Mark Kac posed the question: "Can you hear the shape of a drum?" - meaning, can you reconstruct the geometry of a vibrating membrane from the frequencies it produces? The question turned out to have a negative answer in general, but it crystallized one of the deepest ideas in mathematics: the spectrum of a differential operator encodes geometric information.

Spectral graph theory asks the same question for discrete structures. A graph $G = (V, E)$ has an associated matrix - the Laplacian $L$ - whose eigenvalues form the graph spectrum. These numbers are not arbitrary: they encode whether the graph is connected, how tightly its communities are glued together, how quickly a random walk mixes across its edges, how hard it is to cut the graph in two.

Think of a social network. Each person is a node; each friendship is an edge. The graph has "natural frequencies": a society with two isolated groups (a disconnected graph) has a different spectrum from one that is fully interconnected. The small eigenvalues of $L$ correspond to smooth, slowly-varying signals - the overall community membership function. The large eigenvalues correspond to rapidly-oscillating signals - the microscopic variation from person to person. This is the graph analogue of low and high frequencies in audio.

Three statements, each surprising when first encountered, that spectral graph theory makes precise:

  1. The number of connected components of $G$ equals the number of times $0$ appears as an eigenvalue of $L$.
  2. The second-smallest eigenvalue $\lambda_2(L)$ - the "Fiedler value" - tells you how hard it is to disconnect the graph. A graph is harder to cut when $\lambda_2$ is larger.
  3. The eigenvector corresponding to $\lambda_2$ - the "Fiedler vector" - assigns a real number to each vertex, and the sign of this number tells you which side of the best bisection each vertex belongs to.

These are not vague analogies. They are theorems with proofs, and they form the backbone of a theory that has become indispensable in machine learning.

1.2 The Three Graph Matrices

For a graph $G = (V, E)$ with $n = |V|$ vertices and $m = |E|$ edges, three matrices appear constantly:

Adjacency matrix $A \in \mathbb{R}^{n \times n}$:

$$A_{ij} = $$

For undirected graphs, $A$ is symmetric. For weighted graphs, $A_{ij} = w_{ij}$, the edge weight. The adjacency matrix encodes the direct connections in the graph.

Degree matrix $D \in \mathbb{R}^{n \times n}$: a diagonal matrix with $D_{ii} = d_i = \sum_j A_{ij}$, the degree of vertex $i$. For weighted graphs, $d_i = \sum_j w_{ij}$ is the weighted degree (also called strength).

Graph Laplacian $L = D - A$: the central object of spectral graph theory. Explicitly:

$$L_{ij} = $$

The Laplacian is named after Pierre-Simon Laplace because it is the discrete analogue of the continuous Laplace operator $\Delta = \sum_i \partial^2/\partial x_i^2$. For a function $f: V \to \mathbb{R}$ defined on the vertices:

$$(Lf)_i = \sum_{j: (i,j) \in E} (f(i) - f(j)) = d_i f(i) - \sum_{j: (i,j) \in E} f(j)$$

This is the "discrete second derivative" - it measures how much the value at vertex $i$ differs from the average value among its neighbors.

For AI: In a Graph Neural Network, the operation $\tilde{A} \mathbf{H}$ (multiplying node features by the adjacency matrix with self-loops) is equivalent to computing $(\tilde{D} - \tilde{L})\mathbf{H}$. The Laplacian is implicitly present in every GNN layer.

1.3 Why Eigenvalues Reveal Structure

The Laplacian $L$ is a real symmetric positive semidefinite matrix. By the spectral theorem (03-Advanced-Linear-Algebra), it has a complete orthonormal basis of eigenvectors $\mathbf{u}_1, \mathbf{u}_2, \ldots, \mathbf{u}_n$ with real non-negative eigenvalues:

$$0 = \lambda_1 \leq \lambda_2 \leq \cdots \leq \lambda_n$$

Why is $\lambda_1 = 0$ always? Because $L \mathbf{1} = \mathbf{0}$ - the all-ones vector is always in the null space of $L$ (every row of $L$ sums to zero). The constant function "assign the same value to every vertex" has zero variation across every edge, so it has zero energy.

The deeper result: $\lambda_1 = \lambda_2 = \cdots = \lambda_k = 0$ if and only if the graph has exactly $k$ connected components. On a disconnected graph with $k$ components, the eigenvectors for eigenvalue $0$ are the indicator vectors of each component.

The first nonzero eigenvalue $\lambda_2$ - if it exists - is called the algebraic connectivity or Fiedler value (after Miroslav Fiedler, who proved its key properties in 1973). A larger $\lambda_2$ means the graph is harder to disconnect; a $\lambda_2$ close to zero means there is almost a disconnection - a bottleneck.

The largest eigenvalue $\lambda_n$ gives the spectral radius of the Laplacian and satisfies $\lambda_n \leq 2 d_{\max}$.

1.4 Historical Timeline

SPECTRAL GRAPH THEORY - HISTORICAL TIMELINE
========================================================================

  1847  Kirchhoff     - Matrix-Tree theorem; Laplacian for electrical circuits
  1931  Whitney       - Graph isomorphism; chromatic polynomials
  1954  Collatz &     - Systematic study of graph spectra begins
        Sinogowitz
  1973  Fiedler       - Algebraic connectivity; Fiedler vector; graph bisection
  1981  Cheeger       - Cheeger inequality (originally for manifolds)
  1988  Alon & Milman - Discrete Cheeger inequality for graphs
  1996  Belkin &      - Laplacian eigenmaps for manifold learning (pub. 2001)
        Niyogi
  2000  Shi & Malik   - Normalized Cuts and image segmentation
  2002  Ng, Jordan,   - Spectral clustering algorithm (the standard version)
        Weiss
  2004  Spielman &    - Spectral sparsification; fast Laplacian solvers
        Teng
  2011  Hammond et al - Wavelets on graphs
  2014  Bruna et al   - Spectral graph CNNs (first spectral GNN)
  2016  Defferrard    - ChebNet: Chebyshev polynomial filters on graphs
        et al
  2017  Kipf &        - GCN: first-order Chebyshev -> simple spatial rule
        Welling
  2022  Dwivedi et al - Laplacian positional encodings for graph Transformers
  2022  Rampasek et   - GPS: General, Powerful, Scalable graph Transformer
        al                with spectral PE

========================================================================

1.5 Roadmap of the Section

This section follows a deliberate progression from foundational algebra to modern AI applications:

SECTION ROADMAP
========================================================================

  2 Graph Matrices          Build the algebraic objects
         down
  3 Quadratic Form / PSD    Prove fundamental spectral properties
         down
  4 Fiedler Vector          Connect \lambda_2 to graph connectivity
         down
  5 Cheeger Inequality      Connect \lambda_2 to cut structure and mixing
         down
  6 Graph Fourier Transform  Signal processing on graphs
         down
  7 Spectral Filtering      From Fourier to polynomial approximations
         down
  8 Spectral Clustering     Partition graphs via eigenvectors
         down
  9 Laplacian Eigenmaps     Embed graphs; PE for transformers
         down
  10 Directed Graphs        PageRank; complex eigenvalues
         down
  11 Advanced Topics        Sparsification; wavelets; random matrices
         down
  12 ML Applications        KGs, molecules, LLM attention analysis

========================================================================

2. Graph Matrices and Their Spectra

2.1 Adjacency Matrix: Spectral View

Definition. For $G = (V, E)$ with $n$ vertices, the adjacency matrix $A \in \mathbb{R}^{n \times n}$ is defined by $A_{ij} = w_{ij}$ if $(i,j) \in E$ and $A_{ij} = 0$ otherwise (with $w_{ij} = 1$ for unweighted graphs).

Key spectral property: Walk counting. The $(i,j)$ entry of $A^k$ counts the number of walks of length exactly $k$ from vertex $i$ to vertex $j$. This follows by induction: $(A^k)_{ij} = \sum_\ell (A^{k-1})_{i\ell} A_{\ell j}$ sums over all ways to reach $j$ in $k$ steps by first taking $k-1$ steps to reach $\ell$, then one step to $j$.

For AI: This walk-counting property is the spectral justification for why a $k$-layer GNN can "see" information from the $k$-hop neighborhood. The matrix $A^k$ is what a linear GNN with $k$ layers computes.

Eigenvalues of $A$. For an undirected graph, $A$ is symmetric, so all eigenvalues are real. Let $\mu_1 \geq \mu_2 \geq \cdots \geq \mu_n$ denote the eigenvalues of $A$ in decreasing order. Key bounds:

  • For any graph: $|\mu_i| \leq d_{\max}$ (the maximum degree), since $\lVert A \rVert_2 = \mu_1 \leq d_{\max}$.
  • For a $d$-regular graph: $\mu_1 = d$ with eigenvector $\mathbf{1}/\sqrt{n}$.
  • Bipartite graphs have symmetric spectra: $\mu_i = -\mu_{n+1-i}$.
  • The number of distinct eigenvalues is at least $\text{diam}(G) + 1$ (where $\text{diam}$ is graph diameter).

Non-examples of symmetry: For a directed graph, $A$ is not symmetric and eigenvalues may be complex. This is why directed spectral theory (10) requires separate treatment.

2.2 Degree Matrix and Volume

The degree matrix $D = \operatorname{diag}(d_1, d_2, \ldots, d_n)$ is diagonal with $D_{ii} = d_i = \sum_j A_{ij}$.

Volume. For a subset $S \subseteq V$, the volume is $\operatorname{vol}(S) = \sum_{i \in S} d_i$. For the full graph, $\operatorname{vol}(V) = 2m$ (each edge contributes 2 to the total degree sum). Volume plays the role of "mass" in the normalized Laplacian theory.

For a $d$-regular graph, $D = d \cdot I_n$ and $\operatorname{vol}(S) = d |S|$, making the theory particularly clean. Most derivations proceed with general $D$ but reduce to simpler formulas in the regular case.

Random walk transition matrix. The matrix $P = D^{-1} A$ is a row-stochastic matrix: $\sum_j P_{ij} = 1$ for all $i$. It defines a random walk on the graph: from vertex $i$, move to neighbor $j$ with probability $A_{ij}/d_i$. The stationary distribution of this walk is $\boldsymbol{\pi}$ with $\pi_i = d_i / \operatorname{vol}(V)$ - proportional to degree. This connection between $D$, $A$, and random walks is central to the normalized Laplacian theory.

2.3 Unnormalized Laplacian L = D - A

Definition. $L = D - A$. Entry-by-entry:

$$L_{ij} = $$

Every row (and column) sums to zero: $\sum_j L_{ij} = d_i - \sum_{j:(i,j)\in E} 1 = 0$. Equivalently, $L \mathbf{1} = \mathbf{0}$.

The fundamental quadratic form:

$$\mathbf{x}^\top L \mathbf{x} = \sum_{(i,j) \in E} w_{ij}(x_i - x_j)^2 \quad \text{for all } \mathbf{x} \in \mathbb{R}^n$$

Proof:

$$\mathbf{x}^\top L \mathbf{x} = \mathbf{x}^\top D \mathbf{x} - \mathbf{x}^\top A \mathbf{x}$$ $$= \sum_i d_i x_i^2 - \sum_{(i,j)\in E} 2 w_{ij} x_i x_j$$ $$= \sum_{(i,j)\in E} w_{ij}(x_i^2 + x_j^2) - 2\sum_{(i,j)\in E} w_{ij} x_i x_j = \sum_{(i,j)\in E} w_{ij}(x_i - x_j)^2 \geq 0$$

Since $w_{ij} > 0$, this is always non-negative: $L \succeq 0$.

Geometric meaning: $\mathbf{x}^\top L \mathbf{x}$ measures the total variation of the signal $\mathbf{x}: V \to \mathbb{R}$ across all edges. It is zero if and only if $x_i = x_j$ for all edges $(i,j)$ - i.e., $\mathbf{x}$ is constant on each connected component.

For AI: Graph regularization in semi-supervised learning minimizes $\mathbf{f}^\top L \mathbf{f}$ subject to labeling constraints. This penalizes label functions that change rapidly across edges - a smoothness prior that says "connected nodes likely have the same label."

2.4 Normalized Laplacians

Two normalized variants of the Laplacian are used in practice:

Symmetric normalized Laplacian:

$$L_{\text{sym}} = D^{-1/2} L D^{-1/2} = I - D^{-1/2} A D^{-1/2}$$

with entries:

$$\left(L_{\text{sym}}\right)_{ij} = $$

Properties: Symmetric; eigenvalues in $[0, 2]$; $\lambda_k(L_{\text{sym}}) = 1$ for all $k$ iff the graph is bipartite (eigenvalues symmetric around 1); $\tilde{\mathbf{u}}_k = D^{1/2} \mathbf{u}_k$ where $\mathbf{u}_k$ are eigenvectors of $L_{\text{rw}}$.

Random-walk normalized Laplacian:

$$L_{\text{rw}} = D^{-1} L = I - D^{-1} A = I - P$$

with $P = D^{-1}A$ the random walk transition matrix. Properties: Not symmetric, but has the same eigenvalues as $L_{\text{sym}}$ (they are similar matrices). Eigenvalues in $[0, 2]$. The eigenvectors of $L_{\text{rw}}$ for eigenvalue $0$ are the constant vectors on each component.

When to use which:

Laplacian Use case Why
$L = D - A$ Graphs with uniform degree; theoretical proofs Simplest form
$L_{\text{sym}}$ Spectral clustering (Ng et al.); GCN normalization Symmetric -> orthogonal eigenvectors
$L_{\text{rw}}$ Random walk analysis; Shi-Malik NCut Direct connection to $P$

For AI (GCN connection): The GCN propagation rule $\hat{A} = \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}$ uses the symmetric normalized adjacency of the graph with self-loops - equivalently, $I - L_{\text{sym}}$ of the augmented graph.

2.5 Spectra of Special Graphs

Closed-form eigenvalues for key graph families provide calibration and test cases:

Complete graph $K_n$: $A = \mathbf{1}\mathbf{1}^\top - I$. Eigenvalues of $A$: $n-1$ (once) and $-1$ ($n-1$ times). Eigenvalues of $L$: $0$ (once) and $n$ ($n-1$ times). The graph is maximally connected: $\lambda_2(L) = n$.

Path graph $P_n$: Vertices $\{1, \ldots, n\}$, edges $\{(i, i+1)\}$. Eigenvalues of $L$:

$$\lambda_k = 2 - 2\cos\!\left(\frac{(k-1)\pi}{n}\right), \quad k = 1, 2, \ldots, n$$

So $\lambda_1 = 0$, $\lambda_2 = 2 - 2\cos(\pi/n) \approx \pi^2/n^2$ for large $n$ - very small. This reflects the intuition that a long path is easy to cut (just remove the middle edge).

Cycle graph $C_n$: Eigenvalues of $L$:

$$\lambda_k = 2 - 2\cos\!\left(\frac{2\pi(k-1)}{n}\right), \quad k = 1, 2, \ldots, n$$

The spectrum is symmetric around $\lambda = 2$. $\lambda_2 = 2 - 2\cos(2\pi/n) \approx 4\pi^2/n^2$ for large $n$.

Star graph $S_n$: One hub connected to $n-1$ leaves. Eigenvalues of $L$: $0$ (once), $1$ ($n-2$ times), $n$ (once). $\lambda_2 = 1$ regardless of how many leaves there are - the star is easy to disconnect (remove the hub).

$d$-regular bipartite graph $K_{n/2, n/2}$: Eigenvalues of $L$: $0, d, d, \ldots, d, 2d$ with the pattern dictated by the bipartite structure; eigenvalue $2d$ indicates bipartiteness.

2.6 Characteristic Polynomial and Cospectral Graphs

The characteristic polynomial of a graph $G$ is $p_G(\lambda) = \det(\lambda I - A)$. The roots are the eigenvalues of $A$. The coefficients of $p_G$ are spectral invariants: the sum of eigenvalues equals $\operatorname{tr}(A) = 0$ (no self-loops); the sum of squares of eigenvalues equals $\operatorname{tr}(A^2) = 2m$.

Cospectral (isospectral) graphs are graphs with identical characteristic polynomials but non-isomorphic structures. The simplest pair: two graphs on 6 vertices found by Schwenk (1973). Cospectrality shows that the spectrum does not uniquely determine a graph - a fundamental limitation of spectral methods. For graph isomorphism testing, the Weisfeiler-Lehman test (05) captures structure that the spectrum misses.

For AI: The WL-expressiveness hierarchy of GNNs (Xu et al., 2019) parallels this cospectrality result. GNNs based on spectral convolution can distinguish everything the Laplacian spectrum distinguishes - but no more. This is one motivation for higher-order GNNs and attention-based methods.

3. The Fundamental Quadratic Form and PSD Structure

3.1 Dirichlet Energy

The quadratic form $\mathbf{x}^\top L \mathbf{x} = \sum_{(i,j)\in E}(x_i - x_j)^2$ is called the Dirichlet energy (or graph Dirichlet form) of the signal $\mathbf{x}: V \to \mathbb{R}$.

This name comes from the continuous analogue: for a function $f: \Omega \to \mathbb{R}$ on a domain $\Omega \subset \mathbb{R}^d$, the Dirichlet energy is $\int_\Omega \lVert \nabla f \rVert^2 \, d\mathbf{x}$, which measures the total variation (smoothness) of $f$. The graph Laplacian $L$ is the discrete analogue of $-\Delta$ (the negative Laplacian), and $\mathbf{x}^\top L \mathbf{x}$ is the discrete Dirichlet energy.

Interpretations by context:

Context What $\mathbf{x}^\top L \mathbf{x}$ measures
Social network Total disagreement when $x_i \in \{-1, +1\}$ labels communities
Signal on graph Total variation (roughness) of the signal across edges
Temperature field Total heat flux across edges at steady state
Node embeddings "Embedding strain" - how much nearby nodes differ
Semi-supervised labels Penalty for assigning different labels to connected nodes

Critical point of Dirichlet energy. The Rayleigh quotient $R(\mathbf{x}) = \mathbf{x}^\top L \mathbf{x} / \lVert \mathbf{x} \rVert^2$ is minimized by the eigenvector with smallest eigenvalue. Constrained to $\mathbf{x} \perp \mathbf{1}$ (orthogonal to the trivial null vector), the minimum is achieved by $\mathbf{u}_2$, the Fiedler vector.

3.2 Proof That L \succeq 0

Theorem. For any undirected weighted graph with non-negative edge weights, $L \succeq 0$.

Proof. For any $\mathbf{x} \in \mathbb{R}^n$:

$$\mathbf{x}^\top L \mathbf{x} = \sum_{(i,j) \in E} w_{ij}(x_i - x_j)^2 \geq 0$$

since $w_{ij} \geq 0$ and $(x_i - x_j)^2 \geq 0$ for all real numbers. $\square$

Corollary. All eigenvalues of $L$ are non-negative: $0 \leq \lambda_1 \leq \lambda_2 \leq \cdots \leq \lambda_n$.

Corollary. $\mathbf{1}$ is always an eigenvector with $\lambda_1 = 0$, since $L\mathbf{1} = (D - A)\mathbf{1} = \mathbf{d} - A\mathbf{1} = \mathbf{0}$ (where $\mathbf{d}$ is the degree vector, equal to $A\mathbf{1}$).

Strengthened result for normalized Laplacians. For $L_{\text{sym}}$: since $L_{\text{sym}} = D^{-1/2}LD^{-1/2}$ and $L \succeq 0$, we have $L_{\text{sym}} \succeq 0$. Moreover, $\mathbf{x}^\top L_{\text{sym}} \mathbf{x} \leq 2\lVert \mathbf{x} \rVert^2$ for all $\mathbf{x}$, so $\lambda_k(L_{\text{sym}}) \in [0, 2]$.

3.3 Connected Components via the Null Space

Theorem (Fiedler, 1973). The multiplicity of eigenvalue $0$ of the graph Laplacian $L$ equals the number of connected components of $G$.

Proof.

$(\Rightarrow)$

Suppose has connected components . For each component , define as the indicator vector of : if , else . Then because for any vertex :

$$(L\mathbf{v}^\ell)_i = d_i v^\ell_i - \sum_{j:(i,j)\in E} v^\ell_j = d_i - d_i = 0$$

(all neighbors of $i$ are also in $C_\ell$ since components are isolated). The $k$ vectors $\mathbf{v}^1, \ldots, \mathbf{v}^k$ are linearly independent, so $\dim(\ker L) \geq k$.

$(\Leftarrow)$

Suppose . Then , which forces for every edge . Thus is constant on each connected component. The dimension of the space of such functions equals the number of components. So .

Combining both directions, $\dim(\ker L) = k$. $\square$

Examples:

  • Fully connected graph ($k=1$): $\lambda_1 = 0$ is simple; $\lambda_2 > 0$.
  • Graph with 2 isolated components: $\lambda_1 = \lambda_2 = 0$; $\lambda_3 > 0$.
  • Path $P_n$: always connected; $\lambda_2 = 2 - 2\cos(\pi/n) > 0$.

Non-example: For a disconnected graph with components of different sizes, the eigenvectors for $\lambda = 0$ are NOT all-ones vectors but rather indicator vectors of the components.

3.4 Eigenvalue Bounds and Interlacing

Upper bound. For any connected graph:

$$\lambda_n(L) \leq 2 d_{\max}$$

where $d_{\max}$ is the maximum degree. For $d$-regular graphs: $\lambda_n(L) = 2d$ iff the graph is bipartite.

Lower bound on $\lambda_2$. From the Cheeger inequality (full treatment in 5):

$$\lambda_2 \geq \frac{h(G)^2}{2}$$

where $h(G)$ is the Cheeger constant.

Interlacing theorem. Let $H$ be an induced subgraph of $G$ on $m < n$ vertices, with Laplacian eigenvalues $0 = \mu_1 \leq \mu_2 \leq \cdots \leq \mu_m$. Then:

$$\lambda_i(L_G) \leq \lambda_i(L_H) \leq \lambda_{n-m+i}(L_G) \quad \text{for } i = 1, \ldots, m$$

Interlacing means that removing vertices from a graph cannot increase $\lambda_2$ by more than the increase in $\lambda_n$. This is used in structural arguments about graph connectivity after vertex removal.

3.5 Courant-Fischer Minimax Theorem

The Courant-Fischer theorem provides a variational characterization of every eigenvalue of a symmetric matrix. For the graph Laplacian $L$ with eigenvalues $0 = \lambda_1 \leq \lambda_2 \leq \cdots \leq \lambda_n$:

$$\lambda_k = \min_{\substack{S \leq \mathbb{R}^n \\ \dim(S) = k}} \max_{\substack{\mathbf{x} \in S \\ \mathbf{x} \neq \mathbf{0}}} \frac{\mathbf{x}^\top L \mathbf{x}}{\lVert \mathbf{x} \rVert^2}$$

In particular, the Fiedler value has the characterization:

$$\lambda_2 = \min_{\mathbf{x} \perp \mathbf{1},\, \mathbf{x} \neq \mathbf{0}} \frac{\mathbf{x}^\top L \mathbf{x}}{\lVert \mathbf{x} \rVert^2} = \min_{\mathbf{x} \perp \mathbf{1},\, \lVert \mathbf{x} \rVert = 1} \sum_{(i,j)\in E}(x_i - x_j)^2$$

Proof sketch. Write $\mathbf{x} = \sum_k c_k \mathbf{u}_k$ in the eigenbasis. Then $\mathbf{x}^\top L \mathbf{x} = \sum_k \lambda_k c_k^2$ and $\lVert \mathbf{x} \rVert^2 = \sum_k c_k^2$. The Rayleigh quotient is $\sum_k \lambda_k c_k^2 / \sum_k c_k^2$, a convex combination of eigenvalues. Minimizing over $\mathbf{x} \perp \mathbf{u}_1 = \mathbf{1}/\sqrt{n}$ forces $c_1 = 0$, making the minimum $\lambda_2$ (achieved when $c_2 = 1$, all others $0$).

Practical use. Courant-Fischer justifies using $\mathbf{u}_2$ as the optimal graph bisection vector: it solves the continuous relaxation of the minimum bisection problem, as we prove in 4 and 8.

4. Algebraic Connectivity and the Fiedler Vector

4.1 Algebraic Connectivity \lambda_2

Definition. The algebraic connectivity of a graph $G$ is $a(G) = \lambda_2(L)$, the second-smallest eigenvalue of the graph Laplacian. It is also called the Fiedler value.

Theorem (Fiedler, 1973). $\lambda_2(L) > 0$ if and only if $G$ is connected.

This follows directly from 3.3: $\lambda_2 = 0$ iff there are at least 2 connected components.

Why "algebraic" connectivity? The classical combinatorial connectivity $\kappa(G)$ (minimum number of vertices whose removal disconnects $G$) is NP-hard to compute in general. The algebraic connectivity $\lambda_2$ provides a computable lower bound:

$$\lambda_2 \leq \kappa(G) \leq \delta(G)$$

where $\delta(G)$ is the minimum degree. This inequality chain says: algebraic connectivity $\leq$ vertex connectivity $\leq$ minimum degree.

Sensitivity. When a single edge $(u,v)$ is added to a graph, $\lambda_2$ can increase by at most $2$. When an edge is removed, $\lambda_2$ can decrease by at most $2$. This quantifies how much the connectivity changes with each graph edit - useful in robust network design.

Regular graphs. For a $d$-regular graph on $n$ vertices:

$$\lambda_2(L) = d - \mu_1(A) \quad \text{(second eigenvalue of adjacency)}$$

where $\mu_1(A)$ is the largest eigenvalue of $A$ not equal to $d$. The spectral gap $d - \mu_1$ of the adjacency matrix and the algebraic connectivity are directly related for regular graphs.

4.2 The Fiedler Vector

Definition. The Fiedler vector $\mathbf{u}_2$ is the eigenvector of $L$ corresponding to $\lambda_2$.

The Fiedler vector assigns a real number $(\mathbf{u}_2)_i$ to each vertex $i$. Vertices with positive values are assigned to one "side" of the graph; vertices with negative values to the other. This is the basis of spectral bisection.

Spectral bisection algorithm: 1. Compute the Fiedler vector $\mathbf{u}_2$. 2. Partition $V = S \cup \bar{S}$ by the sign of $(\mathbf{u}_2)_i$: let $S = \{i : (\mathbf{u}_2)_i \geq 0\}$. 3. The edges $(S, \bar{S})$ form the "spectral cut."

Why does this work? The Courant-Fischer theorem says $\mathbf{u}_2$ minimizes the Dirichlet energy $\sum_{(i,j)\in E}(x_i - x_j)^2$ subject to $\mathbf{x} \perp \mathbf{1}$ and $\lVert \mathbf{x} \rVert = 1$. If we further constrain $x_i \in \{-c, +c\}$ (a discrete two-way partition), we get the NP-hard graph bisection problem. The Fiedler vector is the continuous relaxation of this discrete problem - the best we can do efficiently.

The ordering property. Sorting vertices by their Fiedler vector value $(\mathbf{u}_2)_i$ reveals the community structure of the graph. Vertices in the same community tend to have similar $(\mathbf{u}_2)_i$ values; the transition from negative to positive marks the community boundary.

For AI: Spectral bisection is used in: - Circuit partitioning (VLSI design): split a circuit graph across two chips to minimize inter-chip connections - Domain decomposition (PDE solvers): partition a mesh graph for parallel computation - Community detection in knowledge graphs: find the two most separated communities in a KG

4.3 Bounding Graph Properties via \lambda_2

Diameter bound (Mohar, 1991):

$$\operatorname{diam}(G) \leq \left\lfloor \frac{2\ln(n-1)}{\ln\!\left(\frac{\lambda_n}{\lambda_n - \lambda_2}\right)} \right\rfloor$$

A simpler bound: $\operatorname{diam}(G) \leq \frac{2n}{\lambda_2}$ (rough but useful).

Vertex connectivity. For any connected graph:

$$\lambda_2(L) \leq \kappa(G)$$

where $\kappa(G)$ is the vertex connectivity (minimum number of vertices to remove to disconnect). A large $\lambda_2$ means the graph is robustly connected.

Conductance. The conductance $\Phi(G) = \min_{S: \operatorname{vol}(S) \leq \operatorname{vol}(V)/2} \frac{|E(S, \bar{S})|}{\operatorname{vol}(S)}$ measures the minimum normalized cut. The Cheeger inequality (5) gives:

$$\frac{\lambda_2(L_{\text{sym}})}{2} \leq \Phi(G) \leq \sqrt{2\lambda_2(L_{\text{sym}})}$$

Isoperimetric number. The Cheeger constant $h(G)$ (using $|S|$ instead of $\operatorname{vol}(S)$) satisfies the same type of inequality with the unnormalized $\lambda_2$.

4.4 Computing the Fiedler Vector in Practice

For small graphs ($n \leq$ a few thousand), compute all eigenvalues of $L$ directly via dense symmetric eigensolver (scipy.linalg.eigh). The second column of the eigenvector matrix is $\mathbf{u}_2$.

For large sparse graphs, use iterative methods:

Lanczos algorithm: Builds a tridiagonal matrix $T$ from Krylov vectors $\{L\mathbf{v}, L^2\mathbf{v}, \ldots\}$. Converges to extreme eigenvalues fastest. For the Fiedler vector, we need the smallest nonzero eigenvalue, which requires the shift-invert trick: compute the largest eigenvalue of $(L + \epsilon I)^{-1}$ for small $\epsilon$.

Inverse power iteration with deflation: Since $\lambda_1 = 0$ is known, we can deflate it out. Initialize with a random $\mathbf{x} \perp \mathbf{1}$, repeatedly apply $L$ (via sparse matrix-vector product), normalize, and orthogonalize against $\mathbf{1}$. Convergence rate is $(\lambda_2/\lambda_3)^k$ per iteration.

Randomized Nystrom approximation: For graphs with $n > 10^6$, approximate the low-rank spectral structure using randomized sampling of the Laplacian (Spielman & Srivastava, 2011).

4.5 AI Application: Community Detection

Community detection - finding groups of densely interconnected nodes - is one of the most practically important graph problems. Spectral methods are the gold standard for quality guarantees.

Planted partition model. Generate a graph with $k$ communities of size $n/k$ each, intra-community edge probability $p_{\text{in}}$, inter-community probability $p_{\text{out}} \ll p_{\text{in}}$. Spectral bisection recovers the communities exactly when:

$$p_{\text{in}} - p_{\text{out}} > \sqrt{\frac{2\ln n}{n/k} \cdot \frac{1}{\text{SBM gap}}}$$

(This is the information-theoretic threshold for the Stochastic Block Model.)

Knowledge graph clustering. In a knowledge graph (KG) like Freebase or Wikidata, entities form communities by topic (sports, science, politics). The Fiedler vector of the KG adjacency graph separates these clusters. The resulting community structure can be used to create topic-specific sub-KGs for retrieval-augmented generation.

5. Cheeger's Inequality and Graph Expansion

5.1 The Cheeger Constant h(G)

Definition. For a graph $G = (V, E)$ and a subset $S \subseteq V$, the edge boundary $\partial S$ is the set of edges between $S$ and its complement $\bar{S} = V \setminus S$:

$$\partial S = \{(i,j) \in E : i \in S, j \in \bar{S}\}$$

The conductance (or isoperimetric ratio) of the cut $(S, \bar{S})$ is:

$$\Phi(S) = \frac{|\partial S|}{\min(|S|, |\bar{S}|)} \quad \text{(unnormalized)} \quad \text{or} \quad h(S) = \frac{|\partial S|}{\min(\operatorname{vol}(S), \operatorname{vol}(\bar{S}))} \quad \text{(normalized)}$$

The Cheeger constant (or isoperimetric number) of $G$ is:

$$h(G) = \min_{S \subset V,\, S \neq \emptyset,\, V} h(S)$$

This is the minimum conductance cut: the partition that minimizes the fraction of edges leaving the smaller side relative to its volume. A small $h(G)$ means the graph has a bottleneck - a small number of edges separating a large fraction of the volume.

Computing $h(G)$ is NP-hard. This is a major motivation for the Cheeger inequality, which gives a polynomial-time algorithm (via $\lambda_2$) to find a cut within a factor of $\sqrt{2}$ of optimal.

Examples: - Path $P_n$: Remove the middle edge; $h(P_n) \approx 2/n$. Very small: the path is a severe bottleneck. - Complete graph $K_n$: Every cut $(S, \bar{S})$ has $|\partial S| = |S| \cdot |\bar{S}|/(n-1) \approx |S|/2$; $h(K_n) = n/(2(n-1)) \approx 1/2$. - Expander graph (5.3): $h(G) = \Omega(1)$ - bounded below by a constant, independent of $n$.

5.2 Cheeger's Inequality

Theorem (Alon & Milman, 1985; Dodziuk, 1984). For any undirected graph $G$:

$$\frac{\lambda_2(L_{\text{sym}})}{2} \leq h(G) \leq \sqrt{2\lambda_2(L_{\text{sym}})}$$

Proof of the left inequality (easy direction). We show $\lambda_2 \leq 2h(G)$ by exhibiting a test vector $\mathbf{x}$ with $R(\mathbf{x}) \leq 2h(G)$.

Let $S^*$ be the optimal Cheeger cut with $h(S^*) = h(G)$. Define:

$$x_i = $$

This $\mathbf{x}$ is orthogonal to the stationary distribution of the random walk (which plays the role of $\mathbf{1}$ for the normalized Laplacian). Then:

$$\mathbf{x}^\top L_{\text{sym}} \mathbf{x} = \sum_{(i,j)\in\partial S^*} \left(\frac{x_i - x_j}{\sqrt{d_i d_j}}\right)^2 \cdot d_i d_j$$

After algebraic simplification using the definition of $h$: $R(\mathbf{x}) \leq 2h(G)$. Since $\lambda_2 = \min R(\mathbf{x})$, we get $\lambda_2 \leq 2h(G)$.

Proof of the right inequality (hard direction). We show $h(G) \leq \sqrt{2\lambda_2}$.

Given the Fiedler vector $\mathbf{u}_2$, sort vertices so $u_{2,1} \leq u_{2,2} \leq \cdots \leq u_{2,n}$. For each threshold $t$, let $S_t = \{i : u_{2,i} \leq t\}$. Consider the sweep over all $n-1$ possible thresholds. By the co-area formula for graphs (a discrete version of the co-area formula in differential geometry), the average conductance of these cuts satisfies:

$$\min_t h(S_t) \leq \frac{\sum_t h(S_t) \cdot \Delta\operatorname{vol}(S_t)}{\sum_t \Delta\operatorname{vol}(S_t)} \leq \sqrt{2\lambda_2}$$

The last step uses the Cauchy-Schwarz inequality together with the fact that $\lambda_2 = R(\mathbf{u}_2)$ is the Rayleigh quotient of the Fiedler vector. $\square$

Tightness. The left bound is tight for expanders (5.3). The right bound is tight for paths and other bottleneck graphs where $\lambda_2 \approx h^2/2$.

Practical implication. Given $\lambda_2$, we know $h(G) \in [\lambda_2/2, \sqrt{2\lambda_2}]$. More importantly, the proof of the right inequality is constructive: the sweep over Fiedler vector thresholds finds a cut with conductance $\leq \sqrt{2\lambda_2}$.

5.3 Expander Graphs

Definition. A family of graphs $\{G_n\}$ is an $(n, d, \lambda)$-expander family if: - Each $G_n$ has $n$ vertices and is $d$-regular - The second eigenvalue of $A$ satisfies $\mu_2(A) \leq \lambda < d$ - The spectral gap $d - \lambda$ is bounded below by a positive constant

Equivalently (by Cheeger): $h(G_n) = \Omega(1)$, i.e., the Cheeger constant is bounded below uniformly in $n$.

Why expanders matter: 1. Communication networks: In a $d$-regular expander on $n$ nodes, any message can be routed between any two nodes in $O(\log n)$ hops, using only $d$ connections per node. This is optimal for constant-degree networks. 2. Error-correcting codes: Expander codes (Sipser & Spielman, 1996) achieve linear-time encoding/decoding of codes close to the Shannon capacity. 3. Derandomization: Expanders provide pseudorandom number generators - random walks on expanders mix in $O(\log n)$ steps, so short random walks serve as good randomness sources. 4. GNN depth: A GNN on an expander graph propagates information across the entire graph in $O(\log n)$ layers. This is why expanders are ideal benchmarks for deep GNNs.

Ramanujan graphs. The optimal spectral gap for a $d$-regular graph is bounded: $\lambda \geq 2\sqrt{d-1}$ (Alon-Boppana theorem). Graphs achieving $\lambda = 2\sqrt{d-1}$ are called Ramanujan graphs - they are the optimal expanders. Explicit Ramanujan graph constructions (Lubotzky, Phillips, Sarnak, 1988; Margulis, 1988) use deep number theory.

5.4 Random Walk Mixing Time

The random walk on $G$ defined by transition matrix $P = D^{-1}A$ has stationary distribution $\boldsymbol{\pi}$ with $\pi_i = d_i / \operatorname{vol}(V)$. The mixing time is the number of steps needed for the walk to get close to the stationary distribution:

$$t_{\text{mix}}(\epsilon) = \min\left\{t : \max_i \lVert (P^t)_{i,:} - \boldsymbol{\pi} \rVert_1 \leq \epsilon\right\}$$

Spectral mixing bound. Let $\alpha = \max(|\mu_2(P)|, |\mu_n(P)|) = 1 - \lambda_2(L_{\text{rw}})$ be the second-largest absolute eigenvalue of $P$. Then:

$$t_{\text{mix}}(\epsilon) \leq \frac{\ln(n/\epsilon)}{\lambda_2(L_{\text{rw}})} = \frac{\ln(n/\epsilon)}{1 - \alpha}$$

Interpretation: The spectral gap $\lambda_2 = 1 - \alpha$ governs the mixing time. Large spectral gap -> fast mixing. For expanders with constant spectral gap: $t_{\text{mix}} = O(\log n)$. For paths: $\lambda_2 = O(1/n^2)$, so $t_{\text{mix}} = O(n^2 \log n)$.

Proof sketch. Write the initial distribution as $\boldsymbol{\delta}_i = \boldsymbol{\pi} + \sum_k c_k \boldsymbol{\phi}_k$ where $\boldsymbol{\phi}_k$ are eigenvectors of $P$ (eigenvalues $1 = \mu_1 > \mu_2 \geq \cdots \geq \mu_n \geq -1$). After $t$ steps: $(P^t \boldsymbol{\delta}_i)_j = \pi_j + \sum_{k \geq 2} c_k \mu_k^t (\boldsymbol{\phi}_k)_j$. The deviation decays as $\alpha^t$, giving $t_{\text{mix}} \leq \ln(1/\epsilon\pi_{\min})/\ln(1/\alpha)$.

Lazy walk. To avoid oscillation when $\mu_n \approx -1$ (bipartite-like graphs), use the lazy random walk with $P' = (I + P)/2$. Eigenvalues of $P'$ are $(1 + \mu_k)/2 \in [0, 1]$, avoiding negative eigenvalues.

5.5 AI Connection: Over-Smoothing as Diffusion

Over-smoothing is the well-documented phenomenon in deep GNNs where node representations become indistinguishable as the number of layers increases (Li et al., 2018; Oono & Suzuki, 2020). Spectral theory provides the exact mechanism:

A $k$-layer GCN computes (roughly) $\hat{A}^k X W$ where $\hat{A} = \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}$ is the normalized adjacency. The eigenvalues of $\hat{A}$ satisfy $\hat{\mu}_i = 1 - \tilde{\lambda}_i \in [-1, 1]$ where $\tilde{\lambda}_i$ are eigenvalues of $L_{\text{sym}}$ of the augmented graph. After $k$ iterations:

$$(\hat{A}^k)_{ij} = \sum_\ell \hat{\mu}_\ell^k (U)_{i\ell}(U)_{j\ell} \xrightarrow{k \to \infty} \hat{\mu}_1^k (U)_{i1}(U)_{j1} = \frac{\sqrt{d_i d_j}}{\operatorname{vol}(V)}$$

All node features converge to a value proportional to $\sqrt{d_i}$, determined only by degree - all structural information is lost.

Rate of collapse. The convergence rate is governed by the spectral gap: $\hat{\mu}_2 = 1 - \lambda_2(L_{\text{sym}}) < 1$. Faster collapse on expanders (large $\lambda_2$), slower on bottleneck graphs. This is counterintuitive: the "most connected" graphs (expanders) over-smooth fastest.

Mitigation strategies: - Residual connections (GCNII, Chen et al., 2020): $H^{(k+1)} = \sigma\!\left((1-\alpha)\hat{A}H^{(k)}W^{(k)} + \alpha H^{(0)}\right)$ - preserve initial features - DropEdge (Rong et al., 2020): randomly remove edges during training, reducing effective $k$ - PairNorm (Zhao & Akoglu, 2020): explicitly normalize pairwise distances to prevent collapse - Jumping knowledge (Xu et al., 2018): aggregate representations from all layers

Forward reference: The full architecture-level treatment of over-smoothing, including the WL expressiveness hierarchy and architectural mitigations, is in 11-05 Graph Neural Networks.

6. Graph Fourier Transform and Signal Processing

6.1 Classical Fourier Analogy

The classical Fourier transform on $\mathbb{R}^n$ decomposes a function $f$ into a linear combination of eigenfunctions of the Laplace operator $\Delta$:

$$\hat{f}(\boldsymbol{\omega}) = \int_{\mathbb{R}^n} f(\mathbf{x}) e^{-i\boldsymbol{\omega}^\top \mathbf{x}} d\mathbf{x}$$

The functions $e^{i\boldsymbol{\omega}^\top \mathbf{x}}$ are eigenfunctions of the continuous Laplacian: $-\Delta e^{i\boldsymbol{\omega}^\top \mathbf{x}} = \lVert \boldsymbol{\omega} \rVert^2 e^{i\boldsymbol{\omega}^\top \mathbf{x}}$.

On a graph, the Laplacian $L$ plays the role of $-\Delta$, and its eigenvectors $\mathbf{u}_1, \mathbf{u}_2, \ldots, \mathbf{u}_n$ (with eigenvalues $\lambda_1 \leq \lambda_2 \leq \cdots \leq \lambda_n$) play the role of the complex exponentials $e^{i\boldsymbol{\omega}^\top \mathbf{x}}$.

The analogy:

FOURIER TRANSFORM ANALOGY
========================================================================

  Classical Fourier                   Graph Fourier
  ---------------------------------   ---------------------------------
  Domain          \mathbb{R}^n                  Vertex set V (finite)
  Operator        -\Delta (Laplacian)      L = D - A (graph Laplacian)
  Eigenfunctions  exp(i\omega*x)           Eigenvectors u_1, u_2, ..., u_n
  Frequencies     ||\omega||^2 \in [0, \infty)      Eigenvalues \lambda_1 \leq \lambda_2 \leq ... \leq \lambda_n
  Low freq.       ||\omega|| small -> smooth  \lambda_k small -> smooth on graph
  High freq.      ||\omega|| large -> rapid   \lambda_k large -> rapid variation
  Transform       Continuous integral  Finite matrix multiply (U^Tx)

========================================================================

This analogy is the conceptual foundation for defining convolution, filtering, and signal processing on irregular graph domains.

6.2 Graph Fourier Transform

Definition. Let $L = U \Lambda U^\top$ be the eigendecomposition of the graph Laplacian, with $U = [\mathbf{u}_1, \mathbf{u}_2, \ldots, \mathbf{u}_n]$ the matrix of eigenvectors (columns). For a signal $\mathbf{x} \in \mathbb{R}^n$ (assigning a value $x_i$ to each vertex $i$), the Graph Fourier Transform (GFT) is:

$$\hat{\mathbf{x}} = U^\top \mathbf{x} \in \mathbb{R}^n$$

The inverse GFT is:

$$\mathbf{x} = U \hat{\mathbf{x}} = \sum_{k=1}^n \hat{x}_k \mathbf{u}_k$$

Properties:

  1. Parseval's theorem: $\lVert \hat{\mathbf{x}} \rVert^2 = \lVert \mathbf{x} \rVert^2$ (since $U$ is orthogonal).
  2. Linearity: $\widehat{\mathbf{x} + \mathbf{y}} = \hat{\mathbf{x}} + \hat{\mathbf{y}}$.
  3. Energy decomposition: $\lVert \mathbf{x} \rVert^2 = \sum_k \hat{x}_k^2$ (energy in each frequency component).
  4. Shift property: There is no clean "shift theorem" for graphs as there is for the DFT, because graphs lack translation symmetry. This is a fundamental difference.

Graph convolution. The convolution of two signals $\mathbf{x}$ and $\mathbf{y}$ on a graph is defined spectrally:

$$\mathbf{x} \star_G \mathbf{y} = U (\hat{\mathbf{x}} \odot \hat{\mathbf{y}}) = U \operatorname{diag}(\hat{\mathbf{x}}) \hat{\mathbf{y}}$$

where $\odot$ is element-wise multiplication. This is the analogue of the convolution theorem: convolution in the vertex domain equals pointwise multiplication in the spectral domain.

Limitation of full GFT: Computing $U$ requires $O(n^3)$ time (eigendecomposition). For large graphs ($n > 10^4$), this is infeasible, motivating polynomial approximations (7).

6.3 Frequency Interpretation

The $k$-th frequency component $\hat{x}_k = \langle \mathbf{u}_k, \mathbf{x} \rangle = \mathbf{u}_k^\top \mathbf{x}$ measures how much of the signal $\mathbf{x}$ "oscillates at frequency $\lambda_k$."

Low-frequency signals correspond to small $\lambda_k$: the eigenvectors $\mathbf{u}_k$ for small eigenvalues vary smoothly across edges (since $\mathbf{u}_k^\top L \mathbf{u}_k = \lambda_k$ is small). A signal concentrated in low frequencies is smooth: nearby vertices have similar values.

High-frequency signals correspond to large $\lambda_k$: the eigenvectors for large eigenvalues oscillate rapidly, with $(\mathbf{u}_k)_i$ and $(\mathbf{u}_k)_j$ having opposite signs for many edges $(i,j)$. A pure high-frequency signal looks like a checkerboard on the graph.

Example on a path graph. For $P_n$, the eigenvectors are $u_{k,i} = \sqrt{2/n} \cos((k-1)\pi(2i-1)/(2n))$ - the discrete cosine transform (DCT). The eigenvalues $\lambda_k = 2 - 2\cos((k-1)\pi/n)$ are just the squared DCT frequencies. The GFT on a path is the DCT.

Example on a community graph. A graph with two tightly connected communities has: - $\mathbf{u}_1 = \mathbf{1}/\sqrt{n}$: constant (DC component) - $\mathbf{u}_2$: Fiedler vector, positive on community 1, negative on community 2 - the community membership function is a low-frequency signal - $\mathbf{u}_n$: highest frequency, alternates sign on bipartite-like structure

6.4 Dirichlet Energy Revisited

The Dirichlet energy decomposes cleanly in the spectral domain:

$$\mathbf{x}^\top L \mathbf{x} = \hat{\mathbf{x}}^\top \Lambda \hat{\mathbf{x}} = \sum_{k=1}^n \lambda_k \hat{x}_k^2$$

This is the "power spectrum" interpretation: the Dirichlet energy is the weighted sum of spectral components, weighted by frequency. A signal is smooth (low Dirichlet energy) iff its energy is concentrated in low-frequency components ($\lambda_k$ small).

Spectral analysis of node features. Given a node feature matrix $X \in \mathbb{R}^{n \times d}$, we can compute the spectral content of each feature dimension:

$$\operatorname{Dirichlet}(X_{:,j}) = \sum_{k=1}^n \lambda_k \hat{X}_{kj}^2$$

Feature dimensions with low Dirichlet energy are "community-consistent" features (e.g., political affiliation in a social network). Feature dimensions with high Dirichlet energy are "noisy" local features.

For AI: Graph regularization in semi-supervised learning minimizes:

$$\mathcal{L} = \mathcal{L}_{\text{supervised}} + \gamma \sum_j \mathbf{f}_{:,j}^\top L \mathbf{f}_{:,j}$$

This penalizes high-frequency components in the predicted label function $\mathbf{f}$, implementing a "smoothness prior": connected nodes likely have the same label.

6.5 Uncertainty Principle on Graphs

In classical signal processing, the Heisenberg uncertainty principle states that a signal cannot be simultaneously concentrated in both time and frequency: the product of time spread and frequency spread is at least $1/4\pi$.

On graphs, an analogous uncertainty principle holds (Agaskar & Lu, 2013):

$$\Delta_G(\mathbf{x})^2 \cdot \Delta_S(\mathbf{x})^2 \geq C$$

where $\Delta_G$ measures how localized $\mathbf{x}$ is in the vertex domain (concentrated on a small set of vertices) and $\Delta_S$ measures how localized $\hat{\mathbf{x}}$ is in the spectral domain (concentrated on a small band of frequencies), and $C$ is a constant depending on the graph structure.

Implications for graph signal processing: - A signal perfectly localized on a single vertex ($\Delta_G = 0$) is spread across all frequencies ($\Delta_S = $ maximum) - Smooth signals (concentrated on low frequencies, small $\Delta_S$) are necessarily spread across many vertices ($\Delta_G$ large) - This tradeoff motivates graph wavelets (11.3): basis functions that are approximately localized in both vertex and spectral domains

6.6 AI Application: Node Feature Smoothing

Label propagation (Zhou et al., 2004) is a classic semi-supervised learning algorithm that directly implements low-pass graph filtering. Starting from a partially labeled graph, labels propagate according to:

$$\mathbf{F}^{(t+1)} = \alpha \hat{A} \mathbf{F}^{(t)} + (1 - \alpha) Y$$

where $Y$ is the initial label matrix (zeros for unlabeled nodes), $\hat{A}$ is the normalized adjacency, and $\alpha \in (0,1)$ controls the smoothing strength. In the spectral domain, this converges to:

$$\mathbf{F}^* = (I - \alpha \hat{A})^{-1} (1-\alpha) Y = \sum_k \frac{1-\alpha}{1 - \alpha(1-\tilde{\lambda}_k)} \hat{Y}_k \mathbf{u}_k$$

The filter $g(\lambda) = (1-\alpha)/(1-\alpha(1-\lambda))$ is a low-pass filter: it attenuates high-frequency components ($\lambda$ large) more than low-frequency ones.

For modern LLMs: When an LLM reasons over a knowledge graph, smooth graph signals correspond to consistent facts (nearby entities agree), while high-frequency signals correspond to noise or inconsistencies. Spectral filtering provides a principled way to denoise knowledge graphs before retrieval.

7. Spectral Filtering

7.1 Filtering in the Spectral Domain

A spectral filter on a graph is an operation that modifies the frequency content of a graph signal:

$$\mathbf{y} = g(L)\mathbf{x} = U g(\Lambda) U^\top \mathbf{x}$$

where $g: \mathbb{R} \to \mathbb{R}$ is a scalar function applied pointwise to the eigenvalues: $g(\Lambda) = \operatorname{diag}(g(\lambda_1), g(\lambda_2), \ldots, g(\lambda_n))$.

Common filters:

Filter $g(\lambda)$ Effect AI use case
Low-pass $\mathbf{1}[\lambda \leq \lambda_c]$ Keep low frequencies Smooth node features
High-pass $\mathbf{1}[\lambda > \lambda_c]$ Keep high frequencies Edge detection on graphs
Band-pass $\mathbf{1}[\lambda_a \leq \lambda \leq \lambda_b]$ Keep a frequency band Community detection at scale $k$
Heat kernel $e^{-t\lambda}$ Exponential damping Graph diffusion, PPMI
Identity $1$ No change Trivial
GCN $1 - \lambda/2$ Linear attenuation First-order spectral convolution

Implementation cost: Directly computing $U g(\Lambda) U^\top \mathbf{x}$ requires the full eigendecomposition - $O(n^3)$ preprocessing and $O(n^2)$ per signal. This is intractable for large graphs. Polynomial approximation (7.2) reduces cost to $O(K|E|)$ per signal.

7.2 Polynomial Filters and Localization

A $K$-th order polynomial filter has the form:

$$g(L) = \sum_{k=0}^K \theta_k L^k$$

Key property: K-localization. The filter $g(L) = \sum_{k=0}^K \theta_k L^k$ is exactly $K$-localized: $(g(L)\mathbf{x})_i$ depends only on the values of $\mathbf{x}$ at vertices within graph distance $K$ from $i$.

Proof. $(L^k)_{ij} = 0$ whenever $\text{dist}(i,j) > k$ (by the walk-counting property of graph matrix powers). Therefore $(g(L))_{ij} = \sum_k \theta_k (L^k)_{ij} = 0$ whenever $\text{dist}(i,j) > K$.

Complexity. Computing $\mathbf{y} = g(L)\mathbf{x}$ using the recurrence $L^k \mathbf{x} = L \cdot (L^{k-1}\mathbf{x})$ requires $K$ sparse matrix-vector multiplications, each $O(|E|)$. Total: $O(K|E|)$.

Spatial interpretation. A polynomial filter is exactly equivalent to a $K$-hop neighborhood aggregation, connecting spectral and spatial GNN views. This is the theoretical justification for why GNNs with $K$ layers aggregate information from $K$-hop neighborhoods.

Approximation theorem. By the Stone-Weierstrass theorem, any continuous function $g: [0, \lambda_{\max}] \to \mathbb{R}$ can be uniformly approximated by polynomials. So polynomial filters are universal approximators for spectral filters on any graph.

7.3 Chebyshev Polynomial Approximation

Why Chebyshev? Among all polynomials of degree $\leq K$, the $K$-th Chebyshev polynomial $T_K$ has the smallest maximum deviation from zero on $[-1, 1]$ - it is the optimal polynomial approximation basis.

Definition. The Chebyshev polynomials $T_k: [-1, 1] \to [-1, 1]$ satisfy:

$$T_0(x) = 1, \quad T_1(x) = x, \quad T_k(x) = 2x T_{k-1}(x) - T_{k-2}(x)$$

They have the closed form $T_k(x) = \cos(k \arccos x)$.

Chebyshev graph filter (ChebNet, Defferrard et al., 2016). Scale the Laplacian to $\tilde{L} = 2L/\lambda_{\max} - I \in [-I, I]$ (shifting eigenvalues from $[0, \lambda_{\max}]$ to $[-1, 1]$). Define:

$$g_{\boldsymbol{\theta}}(L) = \sum_{k=0}^K \theta_k T_k(\tilde{L})$$

Computation via recurrence:

$$\bar{\mathbf{x}}_0 = \mathbf{x}, \quad \bar{\mathbf{x}}_1 = \tilde{L}\mathbf{x}, \quad \bar{\mathbf{x}}_k = 2\tilde{L}\bar{\mathbf{x}}_{k-1} - \bar{\mathbf{x}}_{k-2}$$ $$\mathbf{y} = \sum_{k=0}^K \theta_k \bar{\mathbf{x}}_k$$

Each step requires one sparse matrix-vector multiply $O(|E|)$; total cost $O(K|E|)$.

Advantages over truncated Taylor series: - The Chebyshev approximation error decays exponentially in $K$ (geometric convergence for smooth $g$) - No numerical instability from large powers of $\tilde{L}$ - The learned parameters $\theta_k$ have clear frequency interpretation

7.4 Heat Kernel and Diffusion Filters

The graph heat equation generalizes diffusion to graphs:

$$\frac{\partial \mathbf{x}(t)}{\partial t} = -L\mathbf{x}(t), \quad \mathbf{x}(0) = \mathbf{x}_0$$

Solution: $\mathbf{x}(t) = e^{-tL}\mathbf{x}_0$. In the spectral domain: $\hat{x}_k(t) = e^{-t\lambda_k}\hat{x}_{k,0}$ - each frequency decays at rate $\lambda_k$.

The heat kernel $H_t = e^{-tL}$ is a positive semidefinite matrix representing the diffusion of heat on the graph over time $t$. Entries $(H_t)_{ij}$ give the heat at vertex $j$ after time $t$ when a unit heat source is placed at vertex $i$.

Properties: - For $t \to 0$: $H_t \to I$ (no diffusion) - For $t \to \infty$: $H_t \to \frac{1}{n}\mathbf{1}\mathbf{1}^\top$ (heat equalizes, constant temperature on each component) - Relates to random walk: $(H_t)_{ij} = \sum_k e^{-t\lambda_k}(U)_{ik}(U)_{jk}$

Diffusion distance. The distance between vertices $i$ and $j$ at time scale $t$ is:

$$D_t(i,j)^2 = \lVert (H_t)_{i,:} - (H_t)_{j,:} \rVert^2 = \sum_k e^{-2t\lambda_k}(u_{k,i} - u_{k,j})^2$$

This diffusion distance is more robust than shortest-path distance: it accounts for all paths between $i$ and $j$, not just the shortest one.

For AI: The PPMI (Positive Pointwise Mutual Information) matrix used in graph-based word embeddings is approximately a diffusion kernel. The node2vec random walk (Grover & Leskovec, 2016) approximates diffusion distance.

7.5 From Chebyshev to GCN

The GCN layer (Kipf & Welling, 2017) is derived from ChebNet by:

Step 1: Set $K = 1$ (first-order Chebyshev approximation): $g_{\boldsymbol{\theta}}(L) \approx \theta_0 T_0(\tilde{L}) + \theta_1 T_1(\tilde{L}) = \theta_0 I + \theta_1 \tilde{L}$.

Step 2: Approximate $\lambda_{\max} \approx 2$ (holding for regular and near-regular graphs), so $\tilde{L} \approx L - I$.

$$g_{\boldsymbol{\theta}}(L) \approx \theta_0 I + \theta_1(L - I) = \theta_0 I + \theta_1(D - A - I)$$

Step 3: Constrain $\theta = \theta_0 = -\theta_1$ (reduce parameters to prevent overfitting):

$$g_\theta(L) \approx \theta(I + D^{-1/2}AD^{-1/2}) = \theta \hat{A}$$

Step 4: Add self-loops $\tilde{A} = A + I$, renormalize with $\tilde{D}_{ii} = \sum_j \tilde{A}_{ij}$ to prevent numerical issues (the "renormalization trick"):

$$\hat{A} = \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}$$

Full GCN layer:

$$H^{(l+1)} = \sigma\!\left(\tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2} H^{(l)} W^{(l)}\right) = \sigma\!\left(\hat{A} H^{(l)} W^{(l)}\right)$$

Spectral interpretation. The GCN filter $g(\lambda) \approx 1 - \lambda/2$ is a low-pass filter: it passes low frequencies ($\lambda \approx 0$, smooth signals) and attenuates high frequencies ($\lambda \approx 2$, rapidly varying signals). GCN is fundamentally a graph smoother.

Full GNN treatment: For GraphSAGE, GAT, MPNN framework, over-smoothing fixes, and expressiveness theory, see 11-05 Graph Neural Networks.

8. Spectral Clustering

8.1 Graph Partitioning Objectives

Minimum cut. Given a graph and integer $k$, partition $V = A_1 \cup \cdots \cup A_k$ (disjoint, non-empty) to minimize:

$$\text{Cut}(A_1, \ldots, A_k) = \frac{1}{2}\sum_{\ell=1}^k |E(A_\ell, \bar{A}_\ell)|$$

where $E(S, \bar{S})$ is the set of edges between $S$ and its complement.

Problem with minimum cut. Minimum cut tends to cut off isolated vertices or very small sets - the trivial solution $A_1 = \{v\}$ for a low-degree vertex $v$ has very few edges to cut. We need objectives that balance cluster sizes.

RatioCut (Hagen & Kahng, 1992):

$$\text{RatioCut}(A_1, \ldots, A_k) = \sum_{\ell=1}^k \frac{|E(A_\ell, \bar{A}_\ell)|}{|A_\ell|}$$

Normalizes by the number of vertices in each partition - prevents very small cuts.

Normalized Cut (NCut) (Shi & Malik, 2000):

$$\text{NCut}(A_1, \ldots, A_k) = \sum_{\ell=1}^k \frac{|E(A_\ell, \bar{A}_\ell)|}{\operatorname{vol}(A_\ell)}$$

Normalizes by the volume (total degree) - weighted version of RatioCut.

Both problems are NP-hard in general. Spectral clustering relaxes them to tractable eigenvalue problems.

8.2 RatioCut and Unnormalized Spectral Clustering

Two-cluster RatioCut. For $k = 2$ with partition $(S, \bar{S})$:

$$\text{RatioCut}(S, \bar{S}) = |E(S,\bar{S})| \cdot \left(\frac{1}{|S|} + \frac{1}{|\bar{S}|}\right)$$

Define the indicator vector $\mathbf{h} \in \mathbb{R}^n$:

$$h_i = $$

Claim. $\mathbf{h}^\top L \mathbf{h} = n \cdot \text{RatioCut}(S, \bar{S})$. Also: $\lVert \mathbf{h} \rVert^2 = n$ and $\mathbf{h}^\top \mathbf{1} = 0$.

Proof: $\mathbf{h}^\top L \mathbf{h} = \sum_{(i,j)\in E}(h_i - h_j)^2$. The only nonzero terms come from edges crossing the cut: for $(i,j) \in E(S, \bar{S})$:

$$(h_i - h_j)^2 = \left(\sqrt{|\bar{S}|/|S|} + \sqrt{|S|/|\bar{S}|}\right)^2 = \frac{n^2}{|S||\bar{S}|}$$

Summing over all $|E(S,\bar{S})|$ cut edges and using $1/|S| + 1/|\bar{S}| = n/(|S||\bar{S}|)$:

$$\mathbf{h}^\top L \mathbf{h} = |E(S,\bar{S})| \cdot \frac{n^2}{|S||\bar{S}|} = n \cdot \text{RatioCut}(S, \bar{S}) \qquad \square$$

Relaxation. The discrete optimization $\min_S \mathbf{h}^\top L \mathbf{h}$ subject to $\mathbf{h}^\top \mathbf{1} = 0$, $\lVert \mathbf{h} \rVert = \sqrt{n}$, $h_i \in \{c_+, c_-\}$ is NP-hard. Relax the integrality constraint: allow $h_i \in \mathbb{R}$. By Courant-Fischer, the solution is the Fiedler vector $\mathbf{u}_2$.

Recovery. Given $\mathbf{u}_2$, assign vertex $i$ to $S$ if $(\mathbf{u}_2)_i \geq 0$, to $\bar{S}$ otherwise. In practice, use k-means with $k=2$ on $\mathbf{u}_2$ for robustness.

8.3 Normalized Cut (Shi & Malik 2000)

NCut relaxation. Define the indicator $\mathbf{h}$ analogously to RatioCut but with volume weights: for partition $(S, \bar{S})$:

$$h_i = $$

Then $\mathbf{h}^\top L \mathbf{h} = \operatorname{vol}(V) \cdot \text{NCut}(S, \bar{S})$, subject to $\mathbf{h}^\top D \mathbf{1} = 0$ and $\mathbf{h}^\top D \mathbf{h} = \operatorname{vol}(V)$.

Generalized eigenvalue problem. The continuous relaxation is:

$$\min_{\mathbf{h} \perp_D \mathbf{1}} \frac{\mathbf{h}^\top L \mathbf{h}}{\mathbf{h}^\top D \mathbf{h}} = \min_{\tilde{\mathbf{h}} \perp D^{1/2}\mathbf{1}} \frac{\tilde{\mathbf{h}}^\top L_{\text{sym}} \tilde{\mathbf{h}}}{\lVert \tilde{\mathbf{h}} \rVert^2}$$

via the substitution $\tilde{\mathbf{h}} = D^{1/2}\mathbf{h}$. This is the standard Rayleigh quotient for $L_{\text{sym}}$, minimized by $\tilde{\mathbf{u}}_2$. Thus:

$$\mathbf{h}^* = D^{-1/2}\tilde{\mathbf{u}}_2$$

where $\tilde{\mathbf{u}}_2$ is the Fiedler vector of $L_{\text{sym}}$.

Shi-Malik algorithm (2-cluster): 1. Build $L_{\text{sym}} = D^{-1/2}(D-A)D^{-1/2}$ 2. Compute Fiedler vector $\tilde{\mathbf{u}}_2$ of $L_{\text{sym}}$ 3. Assign vertex $i$ to $S$ if $(D^{-1/2}\tilde{\mathbf{u}}_2)_i \geq \text{threshold}$ 4. Choose threshold: empirically (try all $n-1$ thresholds) or at 0

Multi-class NCut. For $k$ clusters, use the $k$ smallest eigenvectors of $L_{\text{sym}}$, form the $n \times k$ matrix $U_k$, normalize each row to unit norm, then apply k-means to the rows.

8.4 Multi-Way Spectral Clustering

The Ng-Jordan-Weiss (NJW) algorithm (2002) is the standard multi-class spectral clustering:

  1. Build the normalized Laplacian $L_{\text{sym}}$
  2. Compute the $k$ smallest eigenvectors $\tilde{\mathbf{u}}_1, \ldots, \tilde{\mathbf{u}}_k$ (smallest eigenvalues of $L_{\text{sym}}$)
  3. Form $U_k \in \mathbb{R}^{n \times k}$ with these eigenvectors as columns
  4. Normalize rows: let $Y_{i,:} = U_{k,i,:} / \lVert U_{k,i,:} \rVert_2$ (row normalization)
  5. Apply k-means to the rows of $Y$

Why row normalization? The perturbation theory of 8.5 shows that in a perfect $k$-cluster graph, the rows of $U_k$ lie exactly on $k$ orthogonal vectors. Row normalization maps these to the same point on the unit sphere regardless of degree, making k-means converge cleanly.

Perturbation theory justification. Consider a "block graph" $G_0$ consisting of $k$ disconnected cliques. The $k$ smallest eigenvalues of $L_{\text{sym}}$ are all $0$, with eigenvectors being the normalized indicators of each clique. Any real graph with $k$ communities can be seen as a perturbed block graph; if the perturbation (inter-community edges) is small, the eigenvectors are close to the block indicators. Weyl's perturbation theorem quantifies how much $\lambda_2$ changes.

8.5 Complete Algorithm and Implementation

SPECTRAL CLUSTERING ALGORITHM
========================================================================

  Input:  Adjacency matrix A \in \mathbb{R}^n^x^n, number of clusters k
  Output: Cluster assignments c \in {1,...,k}^n

  1. Compute degree matrix D = diag(A*1)
  2. Compute normalized Laplacian L_sym = D^(-1/2) (D - A) D^(-1/2)
     (or use L_rw = I - D^(-1) A, but use L_sym for symmetric version)

  3. Compute k smallest eigenvalues and eigenvectors of L_sym
     -> eigenvectors form columns of U_k \in \mathbb{R}^n^x^k

  4. Normalize rows: Y_i = U_k[i,:] / ||U_k[i,:]||_2
     (skip for RatioCut; required for NCut)

  5. Apply k-means clustering to rows of Y
     -> cluster centers \mu_1,...,\mu_k; assignments c[i] \in {1,...,k}

  6. Return c

  Complexity: O(n^3) for full eigendecomposition;
              O(k*n*|E|) with Lanczos + k-means (large graphs)

========================================================================

Practical notes: - Use Lanczos algorithm or LOBPCG for computing the $k$ smallest eigenvectors of $L_{\text{sym}}$ on large sparse graphs (avoid full eigendecomposition) - The choice of $k$ can be guided by the eigengap heuristic: choose $k$ where the gap $\lambda_{k+1} - \lambda_k$ is largest - K-means is run multiple times with random restarts; take the best result (lowest inertia) - For disconnected graphs, the $k$ zero eigenvalues directly give the cluster indicators

8.6 When Spectral Clustering Beats k-Means

K-means minimizes within-cluster variance assuming convex, isotropic, similarly-sized clusters. It fails on non-convex cluster shapes. Spectral clustering has no shape assumption - it works on any cluster structure that is well-separated in the graph.

When spectral clustering excels: - Concentric rings, spirals, moons - any shape detectable by graph connectivity - Clusters at multiple scales (nested communities) - Data with non-Euclidean structure (molecules, social networks)

When k-means excels: - Truly Gaussian clusters in $\mathbb{R}^d$ - Very large $n$ where eigenvector computation is too slow - Cluster structure is well-captured by Euclidean distance

A critical nuance: Spectral clustering requires building the adjacency/affinity graph first. The $k$-NN graph or $\epsilon$-neighborhood graph choice matters enormously for quality. A common failure mode: if the graph is built with too small $k$ or $\epsilon$, communities may become disconnected even within a true cluster. If too large, community structure is washed out.

9. Laplacian Eigenmaps and Graph Embeddings

9.1 The Embedding Problem

Given a graph $G = (V, E)$, we want a mapping $\phi: V \to \mathbb{R}^d$ ($d \ll n$) that preserves the graph structure: vertices that are nearby in the graph should be nearby in the embedding. Formally, we want:

$$\phi = \arg\min_{\phi: V \to \mathbb{R}^d} \sum_{(i,j) \in E} w_{ij} \lVert \phi(i) - \phi(j) \rVert^2$$

subject to constraints that prevent the trivial solution $\phi(i) = \mathbf{0}$ for all $i$.

Decomposing dimension by dimension, this is $d$ separate problems, each of the form:

$$\min_{\mathbf{f} \in \mathbb{R}^n} \mathbf{f}^\top L \mathbf{f} \quad \text{subject to normalization and orthogonality constraints}$$

This is exactly minimizing the Dirichlet energy, solved by the eigenvectors of $L$.

9.2 Laplacian Eigenmaps Algorithm

Belkin & Niyogi (2001/2003). Given $n$ data points $\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(n)} \in \mathbb{R}^D$:

  1. Build the adjacency graph: Connect points $i$ and $j$ if they are among each other's $k$ nearest neighbors (or $\lVert \mathbf{x}^{(i)} - \mathbf{x}^{(j)} \rVert < \epsilon$).

  2. Set edge weights: Use the heat kernel $w_{ij} = \exp(-\lVert \mathbf{x}^{(i)} - \mathbf{x}^{(j)} \rVert^2 / t)$ for connected pairs (with $t > 0$ a bandwidth parameter).

  3. Compute degree and Laplacian: $D_{ii} = \sum_j w_{ij}$, $L = D - W$.

  4. Solve the generalized eigenvalue problem:

$$L \mathbf{u} = \lambda D \mathbf{u}$$

Equivalently: find eigenvectors of $L_{\text{rw}} = D^{-1}L$ (or $L_{\text{sym}}$).

  1. Embed: Take the $d$ eigenvectors $\mathbf{u}_2, \mathbf{u}_3, \ldots, \mathbf{u}_{d+1}$ (skip $\mathbf{u}_1 = \mathbf{1}$) and set $\phi(i) = (u_{2,i}, u_{3,i}, \ldots, u_{d+1,i}) \in \mathbb{R}^d$.

Optimality theorem. The Laplacian eigenmap embedding is the solution to the optimization problem:

$$\min_{\mathbf{f}_1, \ldots, \mathbf{f}_d} \sum_k \mathbf{f}_k^\top L \mathbf{f}_k \quad \text{s.t. } \mathbf{f}_k^\top D \mathbf{f}_k = 1,\; \mathbf{f}_k^\top D \mathbf{f}_j = 0 \text{ for } k \neq j$$

The solution is $\mathbf{f}_k = \mathbf{u}_{k+1}$ (the $(k+1)$-th eigenvector of $L_{\text{sym}}$). This is optimal in the sense that no other $d$-dimensional embedding has smaller total Dirichlet energy.

Manifold learning interpretation. If the data points $\mathbf{x}^{(i)}$ lie on a $d$-dimensional manifold embedded in $\mathbb{R}^D$, the Laplacian eigenmap recovers the intrinsic coordinates of the manifold. As $n \to \infty$ and the bandwidth $t \to 0$ at an appropriate rate, the graph Laplacian converges to the Laplace-Beltrami operator on the manifold (Belkin & Niyogi, 2008).

9.3 Diffusion Maps

Coifman & Lafon (2006) introduced diffusion maps as a multiscale version of Laplacian eigenmaps.

Define the diffusion operator $M = D^{-1}W$ (the random walk matrix) and its $t$-step version $M^t$. The diffusion distance at scale $t$:

$$D_t(i,j)^2 = \sum_k \mu_k^{2t} (\psi_k(i) - \psi_k(j))^2$$

where $\mu_k, \psi_k$ are eigenvalues/eigenvectors of $M$. The diffusion map embedding:

$$\Phi^t(i) = (\mu_2^t \psi_{2,i},\, \mu_3^t \psi_{3,i},\, \ldots,\, \mu_d^t \psi_{d,i}) \in \mathbb{R}^{d-1}$$

The Euclidean distance in the diffusion map equals the diffusion distance: $\lVert \Phi^t(i) - \Phi^t(j) \rVert = D_t(i,j)$.

Multi-scale property. By varying $t$, diffusion maps reveal structure at different scales: - Small $t$: local neighborhood structure - Large $t$: global cluster structure (only the dominant eigenvectors with $\mu_k^t \gg 0$ remain)

9.4 Relationship to PCA and Kernel PCA

Kernel PCA (Scholkopf et al., 1998) computes the principal components of data in a feature space defined by a kernel $k(\mathbf{x}, \mathbf{y})$. For a kernel matrix $K \in \mathbb{R}^{n \times n}$ with $K_{ij} = k(\mathbf{x}^{(i)}, \mathbf{x}^{(j)})$, kernel PCA computes the eigenvectors of the centered kernel matrix.

Commute-time embedding. The commute time $C(i,j)$ between vertices $i$ and $j$ is the expected number of steps for a random walk starting at $i$ to reach $j$ and return. It equals:

$$C(i,j) = \operatorname{vol}(V) \sum_{k=2}^n \frac{1}{\lambda_k}\left(\frac{(u_{k,i})}{\sqrt{d_i}} - \frac{(u_{k,j})}{\sqrt{d_j}}\right)^2$$

This is kernel PCA with the commute-time kernel $k(i,j) = (C(i,i) + C(j,j) - C(i,j))/2$. So Laplacian eigenmaps is a special case of kernel PCA.

9.5 Spectral Positional Encodings for Transformers

Standard Transformers process tokens with positional encodings to handle sequence order. Graph Transformers need analogous positional encodings for graph nodes - but graphs have no canonical ordering.

Laplacian Positional Encoding (LapPE). Use the eigenvectors of the graph Laplacian as node positional encodings:

$$\text{PE}(v) = [\mathbf{u}_2(v), \mathbf{u}_3(v), \ldots, \mathbf{u}_{k+1}(v)] \in \mathbb{R}^k$$

where $\mathbf{u}_j(v)$ is the $v$-th entry of the $j$-th Laplacian eigenvector.

Challenge: Sign ambiguity. Each eigenvector $\mathbf{u}_j$ is defined only up to sign: $-\mathbf{u}_j$ is also a valid eigenvector. This creates non-uniqueness in the PE.

Solutions: - Random sign flips during training (Dwivedi et al., 2022): randomly flip signs in training; the Transformer learns sign-invariant functions - SignNet (Lim et al., 2022): use a Deep Sets architecture that is invariant to sign flips: $\phi(\mathbf{u}_j) + \phi(-\mathbf{u}_j)$ - BasisNet: extend to the case of repeated eigenvalues (multiplicity $> 1$), which introduce rotational ambiguity

RWPE (Random Walk Positional Encoding). Instead of Laplacian eigenvectors, use $k$ steps of a random walk:

$$\text{RWPE}(v)_j = (P^j)_{vv} = \text{probability of returning to } v \text{ after } j \text{ steps}$$

where $P = D^{-1}A$. This avoids the sign ambiguity issue and is invariant to graph automorphisms. Used in GPS (Rampasek et al., 2022) - one of the best-performing graph Transformers.

Why LapPE/RWPE matter. Without positional encodings, graph Transformers cannot distinguish graph structure - all nodes with the same degree distribution look identical. Spectral PE gives each node a unique "spectral fingerprint" derived from its position in the graph's Fourier basis.

10. Directed Graph Spectra

10.1 Directed Laplacians

For a directed graph (digraph) $G = (V, E)$ with $E \subseteq V \times V$ (ordered pairs), the adjacency matrix $A$ is not symmetric: $A_{ij} = 1$ if $(i \to j) \in E$ but $A_{ji}$ may be $0$.

In-degree and out-degree: For each vertex $i$, $d_i^{\text{in}} = \sum_j A_{ji}$ (number of incoming edges) and $d_i^{\text{out}} = \sum_j A_{ij}$ (outgoing edges).

Out-degree Laplacian: $L^{\text{out}} = D^{\text{out}} - A$ where $D^{\text{out}} = \operatorname{diag}(d_1^{\text{out}}, \ldots, d_n^{\text{out}})$.

In-degree Laplacian: $L^{\text{in}} = D^{\text{in}} - A^\top$ (or equivalently, the out-degree Laplacian of the reversed graph).

Key difference from undirected case: - $L^{\text{out}}$ is NOT symmetric in general - Eigenvalues may be complex - The row-sum-zero property holds: $L^{\text{out}}\mathbf{1} = \mathbf{0}$ (since each row sums to $d_i^{\text{out}} - d_i^{\text{out}} = 0$) - But column sums are $d_j^{\text{in}} - d_j^{\text{out}}$, not necessarily zero

Stationary distribution. The directed random walk $P = (D^{\text{out}})^{-1}A$ is row-stochastic. For a strongly connected digraph, the unique stationary distribution $\boldsymbol{\pi}$ satisfies $\boldsymbol{\pi}^\top P = \boldsymbol{\pi}^\top$. The stationary distribution is NOT necessarily uniform (unlike $d$-regular undirected graphs).

10.2 Kirchhoff's Matrix-Tree Theorem

Theorem (Kirchhoff, 1847). For a connected undirected graph $G$, the number of spanning trees equals any cofactor of $L$:

$$\tau(G) = \frac{1}{n}\prod_{k=2}^n \lambda_k(L)$$

where $\lambda_2, \ldots, \lambda_n$ are the non-zero eigenvalues of $L$.

Proof via the Matrix-Tree theorem. By the Matrix-Tree theorem, $\tau(G)$ equals any $(n-1) \times (n-1)$ principal minor of $L$. By the Cauchy-Binet formula, this minor equals $\frac{1}{n}\lambda_2\lambda_3\cdots\lambda_n$, which follows from the spectral decomposition and the fact that $\lambda_1 = 0$ with eigenvector $\mathbf{1}/\sqrt{n}$.

Examples: - $K_n$: $\lambda_2 = \cdots = \lambda_n = n$ (all equal), so $\tau(K_n) = n^{n-2}$ (Cayley's formula). - $P_n$ (path): $\tau(P_n) = 1$ (only one spanning tree - the path itself). - $C_n$ (cycle): $\tau(C_n) = n$.

For AI: The number of spanning trees measures "graph robustness." Networks with many spanning trees (like expanders) remain connected even after many edge failures. This metric appears in network reliability analysis for distributed training clusters.

10.3 PageRank as a Spectral Problem

PageRank (Page, Brin, Motwani, Winograd, 1998) - the algorithm behind Google Search - is fundamentally a spectral computation on a directed graph.

Setup. Model the Web as a directed graph: pages are vertices, hyperlinks are directed edges. Define the Google matrix:

$$G = \alpha P + (1-\alpha) \frac{\mathbf{1}\mathbf{1}^\top}{n}$$

where $P = (D^{\text{out}})^{-1}A$ is the column-stochastic random-walk matrix, $\alpha \in (0,1)$ is the damping factor (typically $\alpha = 0.85$), and $(1-\alpha)\mathbf{1}\mathbf{1}^\top/n$ represents teleportation (random jumps to any page).

PageRank vector. The PageRank of each page is the stationary distribution $\boldsymbol{\pi}$ of the Markov chain defined by $G$:

$$\boldsymbol{\pi}^\top = \boldsymbol{\pi}^\top G \implies \boldsymbol{\pi}^\top(I - G) = \mathbf{0}$$

Equivalently, $\boldsymbol{\pi}$ is the dominant left eigenvector of $G$ (eigenvalue $1$).

Spectral computation. By the Perron-Frobenius theorem, $G$ is a positive stochastic matrix (all entries $> 0$ due to the teleportation term), so it has a unique dominant eigenvalue $\mu_1 = 1$ with a unique positive eigenvector $\boldsymbol{\pi}$.

Power iteration. PageRank is computed by:

$$\boldsymbol{\pi}^{(t+1)} = \boldsymbol{\pi}^{(t)} G = \alpha \boldsymbol{\pi}^{(t)} P + (1-\alpha)\frac{\mathbf{1}}{n}$$

Convergence rate: geometric with ratio $\alpha$ - the second eigenvalue of $G$ is at most $\alpha$. Each iteration is a sparse matrix-vector multiply $O(|E|)$.

For AI (RLHF and LLM preference graphs): In reinforcement learning from human feedback (RLHF), preference data can be modeled as a directed graph over responses, with edge $(r_i, r_j)$ meaning "response $r_i$ is preferred over $r_j$." PageRank on this preference graph gives a global ranking consistent with pairwise preferences. This is closely related to Bradley-Terry models used in reward model training (Ouyang et al., 2022).

10.4 Directed Graph Spectra in AI

Attention as a directed graph. In a Transformer, the attention weights $A_{ij}$ define a directed weighted graph over tokens. The spectral properties of this attention graph have interpretability implications: - The dominant eigenvector of the attention matrix identifies "hub" tokens - tokens that receive most attention - Spectral analysis of attention graphs has been used in mechanistic interpretability to identify "induction heads" and "name mover heads" (Olsson et al., 2022) - The spectral gap of the attention graph determines how quickly information mixes across token positions

Causal DAGs. In causal inference (Chapter 22), structural causal models are represented as DAGs. The spectral properties of the DAG adjacency matrix are related to the "depth" of causal chains: a large spectral radius means long-range causal effects.

11. Advanced Topics

11.1 Spectral Sparsification

Problem. For a dense graph $G$ with $n$ vertices and $\Theta(n^2)$ edges, many spectral algorithms are too slow. Can we find a sparse graph $\tilde{G}$ with $O(n \log n)$ edges that preserves the spectrum of $L$?

Definition. $\tilde{G}$ is an $\epsilon$-spectral sparsifier of $G$ if for all $\mathbf{x} \in \mathbb{R}^n$:

$$(1-\epsilon)\mathbf{x}^\top L_G \mathbf{x} \leq \mathbf{x}^\top L_{\tilde{G}} \mathbf{x} \leq (1+\epsilon)\mathbf{x}^\top L_G \mathbf{x}$$

Equivalently, $(1-\epsilon)L_G \preceq L_{\tilde{G}} \preceq (1+\epsilon)L_G$ in the PSD order.

Theorem (Spielman & Srivastava, 2011). Every graph $G$ has an $\epsilon$-spectral sparsifier with $O(n \log n / \epsilon^2)$ edges, computable in near-linear time using random sampling weighted by effective resistances.

Effective resistance. The effective resistance $R_{\text{eff}}(i,j)$ between vertices $i$ and $j$ is the electrical resistance between them when unit resistors are placed on each edge. It equals:

$$R_{\text{eff}}(i,j) = (\mathbf{e}_i - \mathbf{e}_j)^\top L^\dagger (\mathbf{e}_i - \mathbf{e}_j)$$

where $L^\dagger$ is the pseudoinverse of $L$.

Algorithm: Sample each edge $(i,j)$ independently with probability proportional to $w_{ij} R_{\text{eff}}(i,j)$, and rescale the weight. The resulting sparse graph preserves all spectral properties.

For AI: Spectral sparsification can reduce the computational cost of graph-based ML. A 10-million-edge social graph can be sparsified to $\sim 500K$ edges while preserving spectral clustering quality.

11.2 Random Matrix Theory and Graph Spectra

The spectrum of a random graph has universal limiting behavior described by random matrix theory.

Erdos-Renyi model $G(n, p)$. For a random graph where each edge appears independently with probability $p$, the empirical spectral distribution of $A/\sqrt{np(1-p)}$ converges to the semicircle law (Wigner, 1955):

$$\rho(\lambda) = \frac{1}{2\pi}\sqrt{4 - \lambda^2} \quad \text{for } \lambda \in [-2, 2]$$

The leading eigenvalue separates from the bulk at $\mu_1 \approx np$ when $p \gg \ln n / n$, corresponding to the emergence of a giant connected component.

Implications for GNNs: - Random weight matrices in GNNs have spectra approximated by the semicircle law (for large enough hidden dimensions) - The alignment between the spectra of data graph $A$ and random weight matrices $W$ affects gradient flow in training - Spectral norm regularization of GNN weights controls training stability by constraining the spectral radius

11.3 Graph Wavelets

Motivation. Laplacian eigenvectors are global: $\mathbf{u}_k$ is supported on all $n$ vertices. For signals with local structure (e.g., a signal that varies in one part of the graph but is constant elsewhere), global eigenvectors are inefficient. We need a local, multiscale basis - a graph wavelet transform.

Hammond, Vandergheynst & Gribonval (2011). For a vertex $t$ and scale $s$, define the graph wavelet centered at $t$ at scale $s$:

$$\psi_{s,t} = g_s(L)\boldsymbol{\delta}_t$$

where $\boldsymbol{\delta}_t$ is the indicator vector of vertex $t$ and $g_s(\lambda) = s \cdot g(s\lambda)$ is a scaled spectral filter (bandpass at frequency $1/s$). In the spectral domain:

$$\hat{\psi}_{s,t}(\lambda) = g_s(\lambda) U^\top \boldsymbol{\delta}_t = g_s(\lambda) \mathbf{u}(:,t)$$

Properties of graph wavelets: - Spatially localized: if $g$ has compact spectral support $[\omega_{\min}, \omega_{\max}]$, then $\psi_{s,t}$ is $K$-hop localized where $K$ depends on the bandwidth and $s$ - Frequency selective: wavelets at scale $s$ are sensitive to frequency $\approx 1/s$ - Frame bounds: for appropriate $g$, $\{\psi_{s,t}\}$ forms a frame (redundant but stable basis)

Scattering transform on graphs (Gama et al., 2019): Compose multiple wavelet transforms with pointwise nonlinearities to build invariant/equivariant features. Provides theoretical guarantees for GNN expressiveness.

11.4 Infinite Graphs and Spectral Measures

For infinite graphs (e.g., the integer lattice $\mathbb{Z}^d$, infinite trees), the Laplacian is an unbounded operator on $\ell^2(V)$ and the spectrum is no longer a finite set but a spectral measure $\mu_L$.

Example: Integer lattice $\mathbb{Z}^d$. The Laplacian $L$ on $\mathbb{Z}^d$ has a continuous spectrum $[0, 4d]$ (the $d$-dimensional discrete Laplacian spectrum). This connects to the theory of periodic operators in solid-state physics (Bloch's theorem).

Spectral measure. For a vertex $v$, the spectral measure $\mu_v$ is defined by:

$$\langle \boldsymbol{\delta}_v, f(L) \boldsymbol{\delta}_v \rangle = \int f(\lambda)\, d\mu_v(\lambda)$$

The spectral measure encodes everything about the local geometry of the graph as seen from $v$.

Convergence of finite graphs. If a sequence of finite graphs $G_n$ converges in the Benjamini-Schramm sense to an infinite graph $G_\infty$, the empirical spectral distributions of $L_{G_n}$ converge weakly to the spectral measure of $L_{G_\infty}$.

Preview: The spectral theory of infinite-dimensional operators is the subject of Chapter 12: Functional Analysis, where Hilbert spaces, unbounded operators, and spectral measures are developed fully.

12. Applications in Machine Learning

12.1 Semi-Supervised Learning on Graphs

The problem. Given a graph $G$ with $n$ nodes, a few labeled nodes $\mathcal{L} \subset V$ with labels $y_i \in \{1, \ldots, k\}$, and many unlabeled nodes $\mathcal{U} = V \setminus \mathcal{L}$, assign labels to all unlabeled nodes.

Graph-based regularization (Zhou et al., 2004; Zhu et al., 2003). Find a label function $\mathbf{f}: V \to [0,1]^k$ that: 1. Agrees with the given labels on $\mathcal{L}$ 2. Is smooth on the graph (nearby nodes have similar labels)

The objective:

$$\min_{\mathbf{f}} \sum_{i \in \mathcal{L}} \lVert \mathbf{f}_i - \mathbf{y}_i \rVert^2 + \gamma \sum_{(i,j) \in E} w_{ij} \lVert \mathbf{f}_i - \mathbf{f}_j \rVert^2 = \min_\mathbf{f} \lVert \mathbf{f}_\mathcal{L} - \mathbf{y} \rVert^2 + \gamma \operatorname{tr}(\mathbf{f}^\top L \mathbf{f})$$

The closed-form solution involves $(I + \gamma L)^{-1}$ - a smoothing operator. In the spectral domain:

$$\hat{f}_k = \frac{\hat{y}_k}{1 + \gamma\lambda_k}$$

High-frequency components ($\lambda_k$ large) are strongly regularized toward zero; low-frequency components are preserved.

Connection to label propagation. The Gaussian Fields and Harmonic Functions algorithm (Zhu et al., 2003) sets labeled node values to the true labels and propagates via:

$$\mathbf{f}_\mathcal{U} = L_{\mathcal{U}\mathcal{U}}^{-1} L_{\mathcal{U}\mathcal{L}} \mathbf{y}_\mathcal{L}$$

(where $L_{\mathcal{U}\mathcal{U}}$ is the Laplacian restricted to unlabeled nodes). This harmonic interpolation assigns each unlabeled node the weighted average of its neighbors' labels, with weights determined by graph structure.

Modern variant: GCN for semi-supervised learning. The two-layer GCN of Kipf & Welling (2017) was originally proposed for exactly this task: semi-supervised node classification. The Laplacian smoothing is built into the propagation rule, making it a parameterized (learnable) version of label propagation.

12.2 Knowledge Graph Analysis

A knowledge graph (KG) represents world knowledge as a graph: entities (nodes) connected by typed relations (edges). Examples: Freebase, Wikidata, ConceptNet, UMLS (medical).

Spectral properties of KGs: - KGs are heterogeneous (multiple edge types) and sparse ($|E| = O(|V|)$) - The adjacency spectrum often follows a power law: many small eigenvalues, a few large ones - The spectral gap $\lambda_2$ measures how well-integrated the KG is: a small gap indicates a KG with isolated sub-graphs (different domains not well-connected)

Spectral regularization. KG embedding models (TransE, RotatE, ComplEx) learn entity and relation embeddings. Adding a spectral regularization term:

$$\mathcal{L}_{\text{smooth}} = \sum_r \text{tr}(E_r^\top L_r E_r)$$

encourages entity embeddings to be smooth with respect to each relation type $r$'s graph - entities connected by relation $r$ should have similar embeddings. This improves link prediction accuracy, especially for rare relations.

12.3 Molecular Property Prediction

Molecules are naturally represented as graphs: atoms are nodes, chemical bonds are edges. Predicting molecular properties (solubility, toxicity, drug-likeness) from molecular graphs is a key application of GNNs.

Spectral molecular fingerprints. The eigenvalue spectrum of the molecular graph Laplacian provides rotation- and permutation-invariant descriptors. The "spectral profile" $(\lambda_1, \lambda_2, \ldots, \lambda_n)$ uniquely characterizes many molecular structures.

Graph edit distance and spectral distance. Two molecules have similar properties if they have similar spectral profiles. The distance:

$$d_{\text{spec}}(G_1, G_2) = \lVert \boldsymbol{\lambda}(G_1) - \boldsymbol{\lambda}(G_2) \rVert_2$$

(where eigenvalues are sorted and zero-padded to the same length) approximates graph edit distance and correlates with molecular similarity.

Equivariance and invariance. Spectral fingerprints are invariant to atom permutation (graph isomorphism), which is the correct invariance for molecular property prediction. However, they are blind to chirality (mirror image molecules) - a known limitation requiring higher-order structural features.

12.4 Attention Pattern Analysis in LLMs

A $d$-head attention layer in a Transformer computes $h$ attention weight matrices $\{A^{(1)}, \ldots, A^{(h)}\}$, each $A^{(k)} \in \mathbb{R}^{T \times T}$ for sequence length $T$. These define $h$ weighted directed graphs over the token positions.

Spectral analysis of attention. The eigenvalues of $A^{(k)}$ reveal the attention pattern structure: - If $A^{(k)} \approx \mathbf{1}\mathbf{1}^\top/T$ (uniform attention): $\mu_1 = 1$, all others $\approx 0$ - If $A^{(k)} \approx I$ (attend only to self): $\mu_1 = \cdots = \mu_T = 1$ - Induction heads (Olsson et al., 2022) have $A^{(k)}$ with large spectral gap between $\mu_1$ and $\mu_2$ - they attend sharply to a few positions

Attention graph Laplacian. Define the symmetrized attention Laplacian $L^{(k)} = D^{(k)} - (A^{(k)} + A^{(k)^\top})/2$. The Fiedler vector $\mathbf{u}_2^{(k)}$ of $L^{(k)}$ identifies the two groups of tokens most separated by head $k$'s attention.

Applications: - Attention head pruning: Heads whose attention graphs have very small spectral gap (uniform attention) contribute little and can be pruned (Michel et al., 2019; Voita et al., 2019) - Mechanistic interpretability: Spectral analysis of multi-head attention composition identifies information routing circuits (Elhage et al., 2021) - Context window analysis: The Laplacian spectrum of attention graphs grows as more tokens are added; sudden changes in $\lambda_2$ indicate "phase transitions" in how the model processes context

13. Common Mistakes

# Mistake Why It's Wrong Fix
1 Confusing $L = D - A$ with $L = A - D$ Sign convention: $L \succeq 0$ requires $L = D - A$. With $A - D$, all eigenvalues are $\leq 0$. Always check $L_{ii} = d_i > 0$ (positive diagonal) to verify sign convention
2 Using unnormalized $L$ for spectral clustering on graphs with varying degrees RatioCut (unnormalized) penalizes unequally-sized partitions; for real data, NCut (normalized) gives much better clusters Use $L_{\text{sym}}$ or $L_{\text{rw}}$ for spectral clustering; unnormalized $L$ only for regular graphs
3 Taking the Fiedler vector $\mathbf{u}_1$ (index 1) instead of $\mathbf{u}_2$ $\mathbf{u}_1$ is the constant vector $\mathbf{1}/\sqrt{n}$ (trivial null vector); it has no discriminative information In NumPy: eigenvectors are sorted ascending; take column index 1 (0-indexed), not index 0
4 Ignoring sign ambiguity of eigenvectors For each eigenvector $\mathbf{u}_k$, both $\mathbf{u}_k$ and $-\mathbf{u}_k$ are valid; different runs give different signs Use absolute value $\lvert (\mathbf{u}_k)_i \rvert$ for visualizations; or use RWPE instead of LapPE to avoid sign issues
5 Confusing the Cheeger inequality direction: $\lambda_2 \leq 2h$ vs $h \geq \lambda_2/2$ These are the same inequality; the confusing part is that "large $\lambda_2$" implies "large $h$" (good expander), not "small $h$" Remember: small $\lambda_2$ <-> bottleneck <-> small $h$ <-> easy to cut. Large $\lambda_2$ <-> expander <-> large $h$ <-> hard to cut
6 Computing the graph Laplacian for disconnected graphs and expecting $\lambda_2 > 0$ For a disconnected graph, $\lambda_2 = 0$ always. The null space has dimension equal to the number of components. Check connectivity before spectral clustering. If graph is disconnected, handle each component separately or add a small connectivity term
7 Treating spectral clustering as scale-free (the same regardless of $k$) The cluster structure at scale $k$ uses eigenvectors $\mathbf{u}_2, \ldots, \mathbf{u}_{k+1}$. The $k$-th eigenvector captures increasingly fine-grained structure Choose $k$ using the eigengap heuristic: $k^* = \arg\max_k (\lambda_{k+1} - \lambda_k)$
8 Applying GCN (a low-pass filter) to heterophilic graphs GCN smooths features toward neighborhood averages. For heterophilic graphs (connected nodes have different labels), this destroys discriminative information Use high-pass or band-pass graph filters (e.g., GPRGNN, FAGCN, BernNet) for heterophilic settings
9 Confusing $L_{\text{sym}}$ and $L_{\text{rw}}$: using $L_{\text{rw}}$ for Ng-Jordan-Weiss NJW requires $L_{\text{sym}}$ (symmetric, orthogonal eigenvectors) for row normalization to work. $L_{\text{rw}}$ is not symmetric, so its eigenvectors are not orthogonal Use scipy.linalg.eigh(L_sym) for symmetric eigendecomposition; eigenvectors form orthonormal columns
10 Over-interpreting spectral methods on cospectral graphs Two different graphs can have identical Laplacian spectra. Spectral features cannot distinguish them Augment spectral features with structural features (degree, triangle count, etc.) or use WL-based methods
11 Forgetting the self-loop normalization in GCN derivation Without self-loops, the $K=1$ Chebyshev filter has eigenvalues in $[-1, 1]$; adding $\tilde{A} = A+I$ shifts them to $[0,2]$ Always include self-loops $\tilde{A} = A + I$ and renormalize with $\tilde{D}$ in the GCN propagation rule
12 Using the full GFT ($O(n^3)$) on graphs with $n > 1000$ Full eigendecomposition is $O(n^3)$; for $n = 10^6$, this is $10^{18}$ operations - completely intractable Use polynomial filters (Chebyshev, $K$ sparse matrix-vector products), Lanczos for top-$k$ eigenvectors, or RWPE

14. Exercises

Exercise 1 * - Laplacian Construction and Properties

For the following graph $G$: vertices $\{1, 2, 3, 4\}$, edges $\{(1,2), (2,3), (3,4), (4,1), (1,3)\}$ (an unweighted undirected graph):

(a) Write out $A$, $D$, and $L = D - A$. (b) Verify $\mathbf{x}^\top L \mathbf{x} = \sum_{(i,j)\in E}(x_i - x_j)^2$ for $\mathbf{x} = [1, 2, 0, -1]^\top$. (c) Compute all eigenvalues of $L$. How many connected components does $G$ have? (d) Compute $L_{\text{sym}}$ and verify its eigenvalues lie in $[0, 2]$. (e) For AI: Which eigenvector of $L$ would be used for spectral bisection? What partition does it suggest?

Exercise 2 * - Spectrum of Special Graphs

(a) Derive the eigenvalues of $L$ for the cycle graph $C_5$ (5 vertices in a cycle). Show all work. (b) Compute $\lambda_2(C_5)$. Is the cycle more or less connected (in the algebraic sense) than the path $P_5$? Use the formula for $\lambda_2(P_n)$. (c) For the complete graph $K_n$: prove that $\lambda_2(L_{K_n}) = n$ and all $\lambda_2, \ldots, \lambda_n$ are equal. (d) For a star graph $S_n$ (one hub, $n-1$ leaves): find all eigenvalues and explain geometrically why $\lambda_2 = 1$ regardless of $n$.

Exercise 3 * - Fiedler Vector and Graph Bisection

Given a barbell graph: two cliques $K_5$ connected by a single bridge edge $(u, v)$:

(a) Describe qualitatively what the Fiedler vector looks like without computing it. Which vertices get positive values? Negative? (b) Implement the graph in NumPy, compute $L$, and find $\mathbf{u}_2$ using scipy.linalg.eigh. Plot the Fiedler vector values at each vertex. (c) Use the Fiedler vector to perform spectral bisection. What is $|E(S, \bar{S})|$ for the resulting cut? (d) What is $\lambda_2$ for this graph? Is it close to 0? What does this say about the graph's connectivity?

Exercise 4 ** - Cheeger Inequality Verification

For the path graph $P_{10}$ (10 vertices in a line):

(a) Compute $\lambda_2(L_{\text{sym}})$ analytically using the known formula. (b) Find the Cheeger constant $h(P_{10})$ by enumerating the optimal cut. (Hint: by symmetry, the optimal cut is in the middle.) (c) Verify that Cheeger's inequality $\lambda_2/2 \leq h \leq \sqrt{2\lambda_2}$ holds. How tight are the bounds? (d) Implement the Fiedler vector sweep algorithm to find a cut with conductance $\leq \sqrt{2\lambda_2}$. (e) For AI: If $P_{10}$ were the attention graph of a 10-token sequence, what does the Cheeger constant tell you about information flow?

Exercise 5 ** - Graph Fourier Transform

Define a "community signal" $\mathbf{x}$ on the karate club graph (Zachary 1977, available in NetworkX) where $x_i = +1$ if node $i$ is in community 1 and $x_i = -1$ otherwise.

(a) Compute the GFT $\hat{\mathbf{x}} = U^\top \mathbf{x}$ of the community signal. (b) Plot $|\hat{x}_k|^2$ vs. $\lambda_k$. Is the community signal concentrated in low or high frequencies? (c) Define a "noisy" signal $\mathbf{x}' = \mathbf{x} + 0.5\boldsymbol{\eta}$ where $\boldsymbol{\eta}$ is Gaussian noise. Apply a low-pass filter $g(\lambda) = \mathbf{1}[\lambda \leq \lambda_5]$ to $\mathbf{x}'$ (keep only the first 5 frequency components). (d) Compare the filtered signal to the true community signal. What fraction of nodes are correctly assigned? (e) How does this connect to label propagation in semi-supervised learning?

Exercise 6 ** - Spectral Clustering

Generate a synthetic graph with 3 communities using the Stochastic Block Model: - 3 blocks of 50 nodes each - Intra-block edge probability $p_{\text{in}} = 0.3$, inter-block $p_{\text{out}} = 0.02$

(a) Compute the unnormalized Laplacian $L$. Plot the first 6 eigenvalues. Where is the largest eigengap? (b) Implement the NJW spectral clustering algorithm (Ng-Jordan-Weiss) for $k=3$. (c) Compute the accuracy of the spectral clustering (comparing to the known ground-truth communities, accounting for label permutations). (d) Repeat with $p_{\text{out}} = 0.15$ (near the phase transition). How does the clustering accuracy degrade? (e) For AI: How does the eigengap heuristic perform? Plot accuracy vs. $k$ to verify the correct number of clusters is detected.

Exercise 7 *** - Laplacian Positional Encodings

Implement Laplacian Positional Encodings (LapPE) and test them on a simple graph classification task:

(a) For each graph in a small graph dataset (or a synthetic set with 3 classes: path, cycle, star variants), compute the top-$k$ Laplacian eigenvectors as node features. (b) Handle sign ambiguity by randomly flipping signs of each eigenvector at each forward pass (as in Dwivedi et al., 2022). Show that a model trained this way is sign-invariant. (c) Implement RWPE as an alternative: $\text{RWPE}(v)_j = (P^j)_{vv}$ for $j = 1, \ldots, k$. Compare LapPE and RWPE on the classification task. (d) Explain theoretically why RWPE avoids the sign ambiguity problem while LapPE does not. (e) For AI: In GPT-style attention, can you use RWPE to give the model a "graph-aware" positional encoding? What would this enable for graph reasoning tasks?

Exercise 8 *** - PageRank and Spectral Analysis

Construct a small directed graph representing a citation network (10 papers, edges from citing paper to cited paper):

(a) Implement power iteration to compute the PageRank vector $\boldsymbol{\pi}$ with damping factor $\alpha = 0.85$. Verify convergence. (b) Compute the dominant eigenvalue and eigenvector of the Google matrix $G = \alpha P + (1-\alpha)\mathbf{1}\mathbf{1}^\top/n$ directly via scipy.linalg.eig. Compare to the power iteration result. (c) Add a "dangling node" (a paper with no outgoing citations). How does this affect the Google matrix? How is it handled in practice? (d) Compute the mixing time: how many iterations of power iteration are needed to achieve $\lVert \boldsymbol{\pi}^{(t)} - \boldsymbol{\pi}^* \rVert < 10^{-6}$? How does this relate to $\alpha$? (e) For AI: In RLHF, model responses can be ranked using a directed preference graph. Implement PageRank-based ranking on a set of 5 responses with pairwise preference comparisons. How does it compare to simple win-count ranking?

15. Why This Matters for AI (2026 Perspective)

Concept AI/ML Impact
Graph Laplacian spectrum Foundation of Graph Convolutional Networks (Kipf & Welling, 2017); GCN layer = first-order Chebyshev filter; used in every graph-based ML system
Fiedler vector / spectral bisection Graph partitioning for distributed training (model parallel + pipeline parallel); partition the computation graph of a large model across devices
Cheeger inequality Quantifies over-smoothing rate in deep GNNs; expanders over-smooth fastest; guides choice of GNN depth and skip-connection design
Spectral clustering Gold-standard for community detection in social networks, knowledge graphs, citation networks; used in data curation for LLM pretraining
Graph Fourier Transform Spectral convolution -> polynomial approximation -> spatial GNN: the entire GNN derivation hierarchy is a spectral story; ChebNet (Defferrard et al., 2016)
Laplacian Positional Encodings LapPE in GPS (Rampasek et al., 2022); RWPE in graph Transformers; enables graph Transformers to be position-aware without hardcoded sequence order
Random walk mixing RWPE computation; node2vec walks; GraphSAGE neighborhood sampling; mixing time determines required walk length for meaningful embeddings
Heat kernel / diffusion Graph diffusion networks (Klicpera et al., 2019); APPNP; diffusion-based denoising on knowledge graphs; personalized PageRank for neighborhood aggregation
Spectral sparsification Fast GNN training on large graphs: sparsify the graph while preserving spectral properties; used in GraphSAINT, ClusterGCN
PageRank RLHF preference aggregation; importance weighting in retrieval-augmented generation; entity importance in knowledge graphs
Matrix-Tree theorem Spanning tree sampling for graph augmentation in self-supervised GNN training; tree-structured attention in structured state space models
Directed graph spectra Attention head analysis in mechanistic interpretability (Olsson et al., 2022); causal graph structure in causal LLMs; knowledge graph relation asymmetry

16. Conceptual Bridge

Where we came from. This section builds directly on three pillars: - 11-01 Graph Basics provided the combinatorial vocabulary: vertices, edges, paths, connectivity, bipartiteness. These definitions are the domain of spectral graph theory. - 11-02 Graph Representations introduced the adjacency matrix and Laplacian as data structures. We now treat them as linear operators with rich algebraic structure. - 03 Advanced Linear Algebra (eigenvalues, spectral theorem, PSD matrices) gave us the mathematical machinery. Spectral graph theory is linear algebra applied to graphs.

What this section proved. Starting from the simple definition $L = D - A$, we established: 1. $L \succeq 0$ (proved via the quadratic form $\mathbf{x}^\top L \mathbf{x} = \sum_{(i,j)}(x_i - x_j)^2$) 2. The null space of $L$ encodes connected components 3. $\lambda_2$ quantifies connectivity (Fiedler, 1973) 4. $\lambda_2$ is tightly related to the minimum normalized cut (Cheeger, 1985; Alon-Milman, 1985) 5. Eigenvectors of $L$ form a natural Fourier basis for graph signals 6. Spectral clustering is the continuous relaxation of NP-hard graph partitioning 7. The GCN layer is a first-order spectral filter (Kipf & Welling, 2017)

Where we are going. Two sections lie ahead:

11-05 Graph Neural Networks (immediate next): The spectral foundation developed here - GCN derivation, over-smoothing as diffusion, spectral filters - motivates the full GNN architecture zoo. The MPNN framework, GAT attention, GraphSAGE induction, and over-smoothing mitigations are all seen more clearly through the spectral lens.

12 Functional Analysis (next chapter): The spectral theory of the discrete graph Laplacian is a special case of the spectral theory of self-adjoint operators on Hilbert spaces. The Laplace-Beltrami operator on Riemannian manifolds is the continuous limit of the graph Laplacian. Kernel methods, Mercer's theorem, and Reproducing Kernel Hilbert Spaces (RKHS) generalize what we built here to infinite-dimensional settings.

SPECTRAL GRAPH THEORY IN THE CURRICULUM
============================================================================

  Chapter 02-03           Chapter 11                Chapter 12
  Linear Algebra          Graph Theory              Functional Analysis
  -------------           ------------              --------------------
  Eigenvalues     ------> 04 Spectral       ------> Laplace-Beltrami
  PSD matrices            Graph Theory               operator
  Spectral                     |                    Spectral measure
  theorem                      |                    RKHS
                               |
                    +----------+----------+
                    v                     v
               11-05 GNNs          22 Causal
               GCN, GAT             Inference
               GraphSAGE            Causal DAGs
               MPNN                 d-separation

  KEY RESULTS IN 04:
  ---------------------------------------------------------------------
  L = D - A \succeq 0          (proved via quadratic form)
  ker(L) = span of        (connected components theorem)
  component indicators
  Cheeger: \lambda_2/2 \leq h \leq \sqrt(2\lambda_2)  (connectivity <-> eigenvalue)
  GFT: x = U^Tx          (graph Fourier transform)
  GCN = 1st-order         (from ChebNet K=1, \lambda_max\approx2)
  Chebyshev filter

============================================================================

The unifying theme. Spectral graph theory teaches a single lesson: linear algebraic structure encodes combinatorial structure. The eigenvalues of a matrix you can compute in $O(n^3)$ reveal properties of the graph that are NP-hard to compute directly. This is the power of the spectral approach, and it is why spectral methods remain foundational even as spatial GNNs dominate in practice - the theory explains why the practice works.

$G$$k$$C_1, C_2, \ldots, C_k$$C_\ell$$\mathbf{v}^\ell \in \mathbb{R}^n$$C_\ell$$v^\ell_i = 1$$i \in C_\ell$$0$$L\mathbf{v}^\ell = \mathbf{0}$$i \in C_\ell$$L\mathbf{x} = \mathbf{0}$$0 = \mathbf{x}^\top L \mathbf{x} = \sum_{(i,j)\in E}(x_i - x_j)^2$$x_i = x_j$$(i,j)$$\mathbf{x}$$\dim(\ker L) \leq k$