All Courses
14 · MATH FOR SPECIFIC MODELS

RNN and LSTM Math

Recurrent neural networks model sequences by reusing the same transition at every time step. Their power is memory; their difficulty is training through long chains of time.

Overview

The basic recurrent equation is:

$$ h_t=f_\theta(h_{t-1},x_t). $$

This state update lets a model process variable-length sequences. But training the model means backpropagating through the unrolled computation graph. Long products of recurrent Jacobians can vanish or explode. LSTM and GRU cells add gates and additive memory paths to make sequence learning more stable.

Prerequisites

  • Matrix multiplication and nonlinear activations
  • Chain rule and backpropagation
  • Cross-entropy for sequence prediction
  • Basic transformer attention intuition for the bridge section

Learning Objectives

After this section, you should be able to:

  • Write vanilla RNN, LSTM, and GRU update equations.
  • Explain recurrence as a parameter-shared deep network along time.
  • Derive why gradients vanish or explode through repeated Jacobian products.
  • Apply gradient clipping and truncated BPTT.
  • Interpret LSTM forget/input/output gates and GRU update/reset gates.
  • Distinguish many-to-one, many-to-many, seq2seq, and bidirectional tasks.
  • Explain why attention was introduced to reduce the fixed-context bottleneck.
  • Build diagnostics for masks, gate saturation, length generalization, and gradient norms.

Table of Contents

  1. Sequence Modeling View
  2. 1.1 Sequential data
  3. 1.2 Hidden state
  4. 1.3 Output distribution
  5. 1.4 Autoregressive generation
  6. 1.5 RNN versus transformer
  7. Vanilla RNN Equations
  8. 2.1 State update
  9. 2.2 Output head
  10. 2.3 Parameter sharing
  11. 2.4 Unrolled graph
  12. 2.5 Initial state
  13. Backpropagation Through Time
  14. 3.1 Sequence loss
  15. 3.2 Temporal chain rule
  16. 3.3 Weight gradient
  17. 3.4 Truncated BPTT
  18. 3.5 State detachment
  19. Vanishing and Exploding Gradients
  20. 4.1 Jacobian product
  21. 4.2 Spectral radius
  22. 4.3 Vanishing
  23. 4.4 Exploding
  24. 4.5 Gradient clipping
  25. LSTM Cell
  26. 5.1 Forget gate
  27. 5.2 Input gate
  28. 5.3 Candidate memory
  29. 5.4 Cell update
  30. 5.5 Output gate
  31. GRU Cell
  32. 6.1 Update gate
  33. 6.2 Reset gate
  34. 6.3 Candidate hidden
  35. 6.4 Hidden update
  36. 6.5 GRU versus LSTM
  37. Sequence Tasks
  38. 7.1 Many-to-one
  39. 7.2 Many-to-many
  40. 7.3 Seq2seq
  41. 7.4 Bidirectional RNN
  42. 7.5 Teacher forcing
  43. Attention Bridge
  44. 8.1 Fixed context bottleneck
  45. 8.2 Alignment scores
  46. 8.3 Attention weights
  47. 8.4 Context vector
  48. 8.5 Transformer bridge
  49. Training Practice
  50. 9.1 Padding masks
  51. 9.2 Packed sequences
  52. 9.3 Stateful streaming
  53. 9.4 Initialization
  54. 9.5 Regularization
  55. Diagnostics
  56. 10.1 Shape checks
  57. 10.2 Gradient norms
  58. 10.3 Gate statistics
  59. 10.4 Length tests
  60. 10.5 Ablations

Shape Map

input sequence:       X      shape (B, T, d_x)
hidden state:         h_t    shape (B, d_h)
hidden sequence:      H      shape (B, T, d_h)
logits per token:     O      shape (B, T, |V|)
padding mask:         M      shape (B, T)

1. Sequence Modeling View

This part studies sequence modeling view through the lens of sequence learning. The central question is how information and gradients move through time.

Subtopic Question Formula
Sequential data observations arrive with order and history $x_{1:T}=(x_1,\ldots,x_T)$
Hidden state the model carries a summary of the past $h_t=f_\theta(h_{t-1},x_t)$
Output distribution each state can produce a prediction $p(y_t\mid x_{\le t})=g_\theta(h_t)$
RNN versus transformer RNNs compress history into a state, transformers keep token states visible $h_t$ versus $H_{1:t}$

1.1 Sequential data

Main idea. Observations arrive with order and history.

Core relation:

$$x_{1:T}=(x_1,\ldots,x_T)$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

1.2 Hidden state

Main idea. The model carries a summary of the past.

Core relation:

$$h_t=f_\theta(h_{t-1},x_t)$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

1.3 Output distribution

Main idea. Each state can produce a prediction.

Core relation:

$$p(y_t\mid x_{\le t})=g_\theta(h_t)$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

1.4 Autoregressive generation

Main idea. The previous output can become the next input.

Core relation:

$$p(x_{1:T})=\prod_t p(x_t\mid x_{An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

1.5 RNN versus transformer

Main idea. Rnns compress history into a state, transformers keep token states visible.

Core relation:

$$h_t$ versus $H_{1:t}$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

2. Vanilla RNN Equations

This part studies vanilla rnn equations through the lens of sequence learning. The central question is how information and gradients move through time.

Subtopic Question Formula
Output head map hidden state to logits or regression output $o_t=W_{hy}h_t+b_y$
Parameter sharing the same weights are reused at every time step $\theta_t=\theta$
Unrolled graph an RNN is a deep network along time $h_1\rightarrow h_2\rightarrow\cdots\rightarrow h_T$
Initial state start from zeros or a learned state $h_0=0$ or trainable

2.1 State update

Main idea. Combine current input and previous hidden state.

Core relation:

$$h_t=\phi(W_{xh}x_t+W_{hh}h_{t-1}+b_h)$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

2.2 Output head

Main idea. Map hidden state to logits or regression output.

Core relation:

$$o_t=W_{hy}h_t+b_y$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

2.3 Parameter sharing

Main idea. The same weights are reused at every time step.

Core relation:

$$\theta_t=\theta$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

2.4 Unrolled graph

Main idea. An rnn is a deep network along time.

Core relation:

$$h_1\rightarrow h_2\rightarrow\cdots\rightarrow h_T$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

2.5 Initial state

Main idea. Start from zeros or a learned state.

Core relation:

$$h_0=0$ or trainable$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

3. Backpropagation Through Time

This part studies backpropagation through time through the lens of sequence learning. The central question is how information and gradients move through time.

Subtopic Question Formula
Sequence loss sum or average losses over time $L=\sum_{t=1}^{T}\ell_t$
Temporal chain rule future losses depend on earlier states through recurrence $\partial L/\partial h_t=\partial \ell_t/\partial h_t+(\partial h_{t+1}/\partial h_t)^\top\partial L/\partial h_{t+1}$
Weight gradient shared weights collect gradient from every time step $\partial L/\partial W=\sum_t \partial L_t/\partial W$
Truncated BPTT backpropagate through a limited window for efficiency $t-k,\ldots,t$
State detachment detach hidden state between chunks to control graph length $h_t\leftarrow\mathrm{stopgrad}(h_t)$

3.1 Sequence loss

Main idea. Sum or average losses over time.

Core relation:

$$L=\sum_{t=1}^{T}\ell_t$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

3.2 Temporal chain rule

Main idea. Future losses depend on earlier states through recurrence.

Core relation:

$$\partial L/\partial h_t=\partial \ell_t/\partial h_t+(\partial h_{t+1}/\partial h_t)^\top\partial L/\partial h_{t+1}$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is why RNN training is training a very deep network along time.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

3.3 Weight gradient

Main idea. Shared weights collect gradient from every time step.

Core relation:

$$\partial L/\partial W=\sum_t \partial L_t/\partial W$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

3.4 Truncated BPTT

Main idea. Backpropagate through a limited window for efficiency.

Core relation:

$$t-k,\ldots,t$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

3.5 State detachment

Main idea. Detach hidden state between chunks to control graph length.

Core relation:

$$h_t\leftarrow\mathrm{stopgrad}(h_t)$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

4. Vanishing and Exploding Gradients

This part studies vanishing and exploding gradients through the lens of sequence learning. The central question is how information and gradients move through time.

Subtopic Question Formula
Jacobian product long-range gradients multiply many recurrent Jacobians $\prod_{i=s+1}^{t}\partial h_i/\partial h_{i-1}$
Spectral radius gradient scale depends on recurrent dynamics $\rho(W_{hh})$
Vanishing singular values below one shrink long-range gradients $\Vert g_s\Vert\rightarrow 0$
Exploding singular values above one can blow up gradients $\Vert g_s\Vert\rightarrow\infty$
Gradient clipping cap gradient norm before optimizer update $g\leftarrow g\min(1,c/\Vert g\Vert)$

4.1 Jacobian product

Main idea. Long-range gradients multiply many recurrent jacobians.

Core relation:

$$\prod_{i=s+1}^{t}\partial h_i/\partial h_{i-1}$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

4.2 Spectral radius

Main idea. Gradient scale depends on recurrent dynamics.

Core relation:

$$\rho(W_{hh})$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

4.3 Vanishing

Main idea. Singular values below one shrink long-range gradients.

Core relation:

$$\Vert g_s\Vert\rightarrow 0$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

4.4 Exploding

Main idea. Singular values above one can blow up gradients.

Core relation:

$$\Vert g_s\Vert\rightarrow\infty$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

4.5 Gradient clipping

Main idea. Cap gradient norm before optimizer update.

Core relation:

$$g\leftarrow g\min(1,c/\Vert g\Vert)$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. Clipping became a standard tool because recurrent gradient products can explode suddenly.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

5. LSTM Cell

This part studies lstm cell through the lens of sequence learning. The central question is how information and gradients move through time.

Subtopic Question Formula
Forget gate decide what memory to keep $f_t=\sigma(W_f[x_t,h_{t-1}]+b_f)$
Input gate decide what new content to write $i_t=\sigma(W_i[x_t,h_{t-1}]+b_i)$
Candidate memory propose new cell content $\tilde c_t=\tanh(W_c[x_t,h_{t-1}]+b_c)$
Cell update additive memory path improves gradient flow $c_t=f_t\odot c_{t-1}+i_t\odot\tilde c_t$
Output gate expose part of cell memory as hidden state $h_t=o_t\odot\tanh(c_t)$

5.1 Forget gate

Main idea. Decide what memory to keep.

Core relation:

$$f_t=\sigma(W_f[x_t,h_{t-1}]+b_f)$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

5.2 Input gate

Main idea. Decide what new content to write.

Core relation:

$$i_t=\sigma(W_i[x_t,h_{t-1}]+b_i)$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

5.3 Candidate memory

Main idea. Propose new cell content.

Core relation:

$$\tilde c_t=\tanh(W_c[x_t,h_{t-1}]+b_c)$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

5.4 Cell update

Main idea. Additive memory path improves gradient flow.

Core relation:

$$c_t=f_t\odot c_{t-1}+i_t\odot\tilde c_t$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. The additive cell path is the mathematical reason LSTMs can carry information longer than a plain tanh RNN.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

5.5 Output gate

Main idea. Expose part of cell memory as hidden state.

Core relation:

$$h_t=o_t\odot\tanh(c_t)$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

6. GRU Cell

This part studies gru cell through the lens of sequence learning. The central question is how information and gradients move through time.

Subtopic Question Formula
Update gate interpolate old and candidate states $z_t=\sigma(W_z[x_t,h_{t-1}])$
Reset gate control how much past enters the candidate $r_t=\sigma(W_r[x_t,h_{t-1}])$
Candidate hidden build proposed new hidden state $\tilde h_t=\tanh(W_h[x_t,r_t\odot h_{t-1}])$
Hidden update blend old state and candidate $h_t=(1-z_t)\odot h_{t-1}+z_t\odot\tilde h_t$
GRU versus LSTM GRU is smaller; LSTM has separate cell and hidden state $h_t$ only versus $(c_t,h_t)$

6.1 Update gate

Main idea. Interpolate old and candidate states.

Core relation:

$$z_t=\sigma(W_z[x_t,h_{t-1}])$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

6.2 Reset gate

Main idea. Control how much past enters the candidate.

Core relation:

$$r_t=\sigma(W_r[x_t,h_{t-1}])$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

6.3 Candidate hidden

Main idea. Build proposed new hidden state.

Core relation:

$$\tilde h_t=\tanh(W_h[x_t,r_t\odot h_{t-1}])$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

6.4 Hidden update

Main idea. Blend old state and candidate.

Core relation:

$$h_t=(1-z_t)\odot h_{t-1}+z_t\odot\tilde h_t$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

6.5 GRU versus LSTM

Main idea. Gru is smaller; lstm has separate cell and hidden state.

Core relation:

$$h_t$ only versus $(c_t,h_t)$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

7. Sequence Tasks

This part studies sequence tasks through the lens of sequence learning. The central question is how information and gradients move through time.

Subtopic Question Formula
Many-to-one classify a whole sequence from final or pooled state $\hat y=g(h_T)$
Many-to-many predict at every time step $\hat y_t=g(h_t)$
Seq2seq encode one sequence and decode another $p(y_{1:M}\mid x_{1:T})=\prod_jp(y_j\mid y_{
Bidirectional RNN use past and future context for non-causal tasks $h_t=[\overrightarrow h_t;\overleftarrow h_t]$
### 7.1 Many-to-one

Main idea. Classify a whole sequence from final or pooled state.

Core relation:

$$\hat y=g(h_T)$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

7.2 Many-to-many

Main idea. Predict at every time step.

Core relation:

$$\hat y_t=g(h_t)$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

7.3 Seq2seq

Main idea. Encode one sequence and decode another.

Core relation:

$$p(y_{1:M}\mid x_{1:T})=\prod_jp(y_j\mid y_{An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

7.4 Bidirectional RNN

Main idea. Use past and future context for non-causal tasks.

Core relation:

$$h_t=[\overrightarrow h_t;\overleftarrow h_t]$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

7.5 Teacher forcing

Main idea. Decoder conditions on gold previous outputs during training.

Core relation:

$$p(y_t\mid y_{An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

8. Attention Bridge

This part studies attention bridge through the lens of sequence learning. The central question is how information and gradients move through time.

Subtopic Question Formula
Fixed context bottleneck a single encoder vector struggles with long inputs $c=h_T$
Alignment scores decoder state scores every encoder state $e_{tj}=a(s_{t-1},h_j)$
Attention weights softmax turns scores into a distribution over positions $\alpha_{tj}=\mathrm{softmax}_j(e_{tj})$
Context vector weighted sum exposes relevant encoder states $c_t=\sum_j\alpha_{tj}h_j$
Transformer bridge self-attention removes recurrence and exposes all token states directly $\mathrm{Attention}(Q,K,V)$

8.1 Fixed context bottleneck

Main idea. A single encoder vector struggles with long inputs.

Core relation:

$$c=h_T$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

8.2 Alignment scores

Main idea. Decoder state scores every encoder state.

Core relation:

$$e_{tj}=a(s_{t-1},h_j)$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

8.3 Attention weights

Main idea. Softmax turns scores into a distribution over positions.

Core relation:

$$\alpha_{tj}=\mathrm{softmax}_j(e_{tj})$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is the historical bridge from seq2seq RNNs to transformer attention.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

8.4 Context vector

Main idea. Weighted sum exposes relevant encoder states.

Core relation:

$$c_t=\sum_j\alpha_{tj}h_j$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

8.5 Transformer bridge

Main idea. Self-attention removes recurrence and exposes all token states directly.

Core relation:

$$\mathrm{Attention}(Q,K,V)$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

9. Training Practice

This part studies training practice through the lens of sequence learning. The central question is how information and gradients move through time.

Subtopic Question Formula
Padding masks sequence batches have variable lengths $m_t\in\{0,1\}$
Packed sequences avoid computing loss on padding $L=\sum_tm_t\ell_t/\sum_tm_t$
Initialization orthogonal recurrent matrices can stabilize early dynamics $W_{hh}^\top W_{hh}=I$
Regularization dropout, weight decay, and clipping fight overfit and instability $L+\lambda\Vert\theta\Vert^2$

9.1 Padding masks

Main idea. Sequence batches have variable lengths.

Core relation:

$$m_t\in\{0,1\}$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

9.2 Packed sequences

Main idea. Avoid computing loss on padding.

Core relation:

$$L=\sum_tm_t\ell_t/\sum_tm_t$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

9.3 Stateful streaming

Main idea. Carry state across chunks for long streams.

Core relation:

$$h_\mathrm{next}=h_T$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

9.4 Initialization

Main idea. Orthogonal recurrent matrices can stabilize early dynamics.

Core relation:

$$W_{hh}^\top W_{hh}=I$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

9.5 Regularization

Main idea. Dropout, weight decay, and clipping fight overfit and instability.

Core relation:

$$L+\lambda\Vert\theta\Vert^2$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

10. Diagnostics

This part studies diagnostics through the lens of sequence learning. The central question is how information and gradients move through time.

Subtopic Question Formula
Shape checks inputs, hidden states, gates, and outputs have distinct axes $(B,T,d)$ and $(B,h)$
Gradient norms track exploding or vanishing gradients through time $\Vert g_t\Vert$
Gate statistics saturated gates indicate stuck memory behavior $f_t,i_t,z_t$ near 0 or 1
Length tests evaluate short and long sequences separately $S(T)$
Ablations compare vanilla RNN, GRU, LSTM, and attention bridge $\Delta L,\Delta S$

10.1 Shape checks

Main idea. Inputs, hidden states, gates, and outputs have distinct axes.

Core relation:

$$(B,T,d)$ and $(B,h)$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

10.2 Gradient norms

Main idea. Track exploding or vanishing gradients through time.

Core relation:

$$\Vert g_t\Vert$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

10.3 Gate statistics

Main idea. Saturated gates indicate stuck memory behavior.

Core relation:

$$f_t,i_t,z_t$ near 0 or 1$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. Gate histograms quickly reveal whether an LSTM or GRU is actually using its memory.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

10.4 Length tests

Main idea. Evaluate short and long sequences separately.

Core relation:

$$S(T)$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

10.5 Ablations

Main idea. Compare vanilla rnn, gru, lstm, and attention bridge.

Core relation:

$$\Delta L,\Delta S$$

An RNN is a parameter-shared computation graph unrolled across time. The hidden state is useful because it carries information forward, but it also creates a long chain for gradients to travel backward. LSTM and GRU gates are designed to make this information path more controllable.

Worked micro-example. In a plain linearized RNN, the contribution of an early hidden state to a later hidden state contains powers of the recurrent matrix. If the dominant singular value is 0.8, a signal over 20 steps is scaled by about $0.8^{20}$. If it is 1.2, the same path grows like $1.2^{20}$. This is the vanishing and exploding gradient problem in one line.

Implementation check. For a batch-first tensor, keep the axes explicit: batch, time, feature. A hidden state usually has shape (batch, hidden), while a full output sequence has shape (batch, time, hidden).

AI connection. This is a practical sequence-modeling control variable.

Common mistake. Do not treat the final hidden state as magic memory. For long sequences it can become a bottleneck, which is exactly why attention was introduced in seq2seq systems.

Practice Exercises

  1. Compute one vanilla RNN hidden update.
  2. Compute a sequence log probability from conditional probabilities.
  3. Show a scalar gradient product that vanishes or explodes.
  4. Clip a gradient vector by norm.
  5. Compute one LSTM cell update.
  6. Compute one GRU hidden update.
  7. Apply a padding mask to sequence losses.
  8. Identify shapes for many-to-one and many-to-many tasks.
  9. Compute attention weights and context over encoder states.
  10. Write an RNN debugging checklist.

Why This Matters for AI

Transformers dominate current LLMs, but RNNs still teach the core sequence-learning problems: hidden state, recurrence, long-range credit assignment, gradient stability, teacher forcing, and attention as a solution to fixed-context bottlenecks. Understanding RNNs makes transformer design feel less arbitrary.

Bridge to Transformer Architecture

Transformers replace recurrent state updates with attention over token states. The next section studies how self-attention, residual streams, normalization, and feed-forward blocks solve many RNN bottlenecks while introducing their own memory and compute tradeoffs.

References

  • Sepp Hochreiter and Jurgen Schmidhuber, "Long Short-Term Memory", Neural Computation, 1997: https://doi.org/10.1162/neco.1997.9.8.1735
  • Kyunghyun Cho et al., "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation", 2014: https://arxiv.org/abs/1406.1078
  • Alex Graves, "Generating Sequences With Recurrent Neural Networks", 2013: https://arxiv.org/abs/1308.0850
  • Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio, "On the difficulty of training Recurrent Neural Networks", 2013: https://arxiv.org/abs/1211.5063
  • Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, "Neural Machine Translation by Jointly Learning to Align and Translate", 2014: https://arxiv.org/abs/1409.0473