Batch normalization

Prerequisites

• Standardization: $(x - \mu)/\sigma$.
• Mini-batches.
• Layers and pre-activations.

The analogy

How it works

Formula: for each feature over the batch: $\hat x = (x - \mu_B)/\sqrt{\sigma_B^2 + \varepsilon}$, then $y = \gamma \hat x + \beta$, with learned $\gamma$, $\beta$.

Start with three activations by hand

One neuron's values over a batch of three: $x = [2, 4, 6]$. Predict, then open "Check".

Finish the picture: training versus test

During training the network also keeps running averages, say mean 3.8 and variance 2.5.

• A single test input with value 5. Predict its normalized value with the running statistics.
  Check

  $(5 - 3.8)/\sqrt{2.5} \approx 0.76$ (then scaled by $\gamma$, shifted by $\beta$).

• Why not normalize the test input by its own "batch"? Predict the problem.
  Check

  A batch of one has variance 0, and predictions would depend on which other examples happen to be in the batch. Fixed running statistics make inference deterministic.

Full worked calculation

A mini-batch of 3 examples with two features on very different scales, normalised step by step, then scaled with learned $\gamma, \beta$, and finally the running statistics used at test time.

1. 1Batch mean (per feature)

   $$\mu_1 = \frac{2 + 4 + 6}{3} = 4, \qquad \mu_2 = \frac{10 + 20 + 60}{3} = 30$$

   Batch norm works feature by feature, across the examples of the batch.

2. 2Batch variance

   $$\sigma_1^2 = \frac{(-2)^2 + 0^2 + 2^2}{3} = 2.667, \qquad \sigma_2^2 = \frac{(-20)^2 + (-10)^2 + 30^2}{3} = 466.7$$

   Population variance (divide by $n$). Feature 2 varies about 175 times more than feature 1: without normalisation, the next layer's weights would need very different scales.

3. 3Normalise

   $$\hat x = \frac{x - \mu}{\sqrt{\sigma^2 + \varepsilon}}: \quad \frac{2 - 4}{1.633} = -1.225, \qquad \frac{60 - 30}{21.6} = 1.389$$

   Both features now have mean 0 and standard deviation 1. The small $\varepsilon$ (e.g. $10^{-5}$) avoids dividing by 0 if a feature is constant in the batch.

4. 4Scale and shift

   $$y = \gamma \hat x + \beta, \quad \gamma = 2, \; \beta = 1: \quad [-1.449,\; 1,\; 3.449]$$

   $\gamma$ and $\beta$ are learned like weights. They let the network choose the best mean and spread for each feature, including undoing the normalisation entirely ($\gamma = \sigma$, $\beta = \mu$) if that is best.

5. 5Update the running statistics

   $$\mu_{\text{run}} \leftarrow 0.9\,\mu_{\text{run}} + 0.1 \cdot 4, \qquad \sigma^2_{\text{run}} \leftarrow 0.9\,\sigma^2_{\text{run}} + 0.1 \cdot 2.667$$

   During training, batch norm keeps an exponential average of the batch statistics. After many batches these settle near the dataset's true mean and variance, here about 4 and 2.667.

6. 6Test time: one example, no batch

   $$x = 5: \quad \hat x = \frac{5 - 4}{\sqrt{2.667}} = 0.612, \qquad y = 2 \cdot 0.612 + 1 = 2.225$$

   At test time there may be a single example, so batch statistics make no sense. The running mean and variance are used instead, which makes the prediction for an example independent of the rest of the batch.

The rule in plain words

Training: for each feature, normalize with the current mini-batch's mean and variance, then apply learned $\gamma$ and $\beta$; update running averages of mean and variance.

Inference: use the stored running averages instead of batch statistics.

The code

Read this code after you have normalized the three values by hand. It will make more sense.

void batch_norm(double x[], int n, double gamma, double beta, double out[]) {
    double mean = 0, variance = 0, eps = 1e-5;
    for (int i = 0; i < n; i++) mean += x[i] / n;
    for (int i = 0; i < n; i++) variance += (x[i] - mean) * (x[i] - mean) / n;
    for (int i = 0; i < n; i++)
        out[i] = gamma * (x[i] - mean) / sqrt(variance + eps) + beta;   /* normalize, scale, shift */
}

flowchart LR
  X["layer output x (batch)"] --> M["batch mean, variance"]
  M --> N["x̂ = (x - mean) / √(var + ε)"]
  N --> Y["y = γ x̂ + β"]
  Y --> NEXT["activation / next layer"]
          

Putting numbers on it

The transformation and our numbers:

Common mistakes

• Using the test batch's own statistics at inference.
• Forgetting $\varepsilon$, which divides by zero when the variance is 0.
• Forgetting that $\gamma$ and $\beta$ are learned, not fixed at 1 and 0.

Try it yourself

Did you know?

Key points to remember

If you remember one thing from each line, remember the because.

• Batch norm stabilises training because each layer receives inputs with a consistent scale.
• It needs $\gamma$ and $\beta$ because the network must still be able to choose any scale and offset.
• Inference uses running statistics because predictions shouldn't depend on which examples share a batch.
• Tiny batches break it because their mean and variance are too noisy, or undefined for one example.
• It allows higher learning rates because normalized activations keep gradients well-behaved.

Self-check questions

Answer these in order, out loud. They follow the page, so your answers become the script of your video.

1. How is batch norm like rescaling judges' scores each day?
   Sample answer

   Each day's scores are shifted and scaled to the same average and spread so they can be compared; batch norm does that for each neuron over each mini-batch.

2. Apply batch norm to [2, 4, 6] with $\gamma = 2$, $\beta = 1$.
   Sample answer

   Mean 4, sd 1.63, normalized $[-1.22, 0, 1.22]$, output $[-1.45, 1, 3.45]$.

3. Why does the code add eps inside the square root?

   A good answer mentions zero variance.

   Sample answer

   If all values in the batch are equal, the variance is 0 and the division would blow up; $\varepsilon$ keeps it finite.

4. So what is batch normalization, in one sentence?
   Sample answer

   Normalizing each feature over the mini-batch to zero mean and unit variance, then applying a learned scale and shift.

5. Why does batch norm use running averages at test time?

   A good answer considers a single test example.

   Sample answer

   A single example has no meaningful batch variance, and predictions shouldn't change with the batch composition; running averages from training give fixed, reliable statistics.

6. How many learned parameters does batch norm add per feature?
   Sample answer

   Two, $\gamma$ and $\beta$ (plus two non-learned running statistics).

7. When would you use layer normalization instead?
   Sample answer

   With small batches or in sequence models like Transformers, because it normalizes within each example rather than across the batch.

8. If you had to teach this to a friend in 2 minutes, what would you leave out, and why?
   Sample answer

   I'd leave out the covariate-shift debate and layer norm. I'd keep the judges analogy, the [2, 4, 6] example, and "normalize per batch, learned $\gamma$ and $\beta$, running stats at test time".

External resources

• Ioffe and Szegedy: Batch Normalization (arXiv): the original paper.
• PyTorch docs: nn.BatchNorm1d: parameters, momentum and running statistics.
• Wikipedia: Batch normalization: the method and theories of why it works.

Videos

• YouTube search: batch normalization explained: pick a clear walkthrough.