Bias–variance trade-off

Prerequisites

• MSE.
• Mean and variance.
• The idea of training data versus new (test) data.

The analogy

How it works

Notation: the true relationship is $f(x)$; the data is $y = f(x) + \varepsilon$ with noise $\varepsilon$ of variance $\sigma^2$; $\hat f$ is the model learned from one training sample.

Start with three fits on the same data

Seven points roughly on a U shape (lowest near $x = 3$). Predict, then open "Check".

Finish the picture: bias and variance in numbers

At $x = 2$ the true value is 4. Train five models on five different samples and look at their predictions there.

• Simple models predict 3.0, 3.2, 2.8, 3.1, 2.9. Predict their average, bias² and variance.
  Check

  Average 3.0, so bias² = (3 − 4)² = 1; variance = 0.02. Consistent but wrong.

• Complex models predict 4.8, 3.1, 5.2, 2.9, 4.0. Predict the same three numbers.
  Check

  Average 4.0, so bias² = 0; variance = 0.82. Right on average, but any single model can be far off.

Full worked calculation

The page's experiment at $x = 2$, where the true value is $f(2) = 4$. Five simple models and five complex models were each trained on a different sample. We compute bias² and variance for both, check that they add up to the real error, add noise, and see what averaging the complex models does.

1. 1The average simple model

   $$\mathbb E[\hat f(2)] = \frac{3.0 + 3.2 + 2.8 + 3.1 + 2.9}{5} = 3.0$$

   Averaging over training samples removes the luck of any single dataset. What is left is what the model family systematically predicts.

2. 2Bias² of the simple model

   $$\big(\mathbb E[\hat f(2)] - f(2)\big)^2 = (3.0 - 4)^2 = 1$$

   Even on average it is 1 below the truth. A straight line cannot bend to reach the U shape, and more data would not fix that: this is bias.

3. 3Variance of the simple model

   $$\frac{0^2 + 0.2^2 + (-0.2)^2 + 0.1^2 + (-0.1)^2}{5} = \frac{0.10}{5} = 0.02$$

   Deviations from its own average (3.0), not from the truth. The five predictions are almost identical: the model barely reacts to which sample it saw.

4. 4The complex model: bias²

   $$\mathbb E[\hat f(2)] = \frac{4.8 + 3.1 + 5.2 + 2.9 + 4.0}{5} = 4.0 \;\Rightarrow\; \text{bias}^2 = (4.0 - 4)^2 = 0$$

   On average the flexible model is exactly right. Its shape can follow the true curve.

5. 5The complex model: variance

   $$\frac{0.8^2 + (-0.9)^2 + 1.2^2 + (-1.1)^2 + 0^2}{5} = \frac{4.10}{5} = 0.82$$

   But each single model is far off, in a different direction each time, because it also fits the noise of its own sample.

6. 6Check: bias² + variance = mean squared error

   $$\text{simple: } 1 + 0.02 = 1.02 = \tfrac{1 + 0.64 + 1.44 + 0.81 + 1.21}{5}, \qquad \text{complex: } 0 + 0.82 = 0.82$$

   Computing the error of each model against the truth directly (the table) gives the same numbers. The decomposition is not an approximation: it is exact.

7. 7Add the noise term

   $$\sigma^2 = 0.25 \;\Rightarrow\; \text{expected error on new } y: \quad 1.02 + 0.25 = 1.27, \qquad 0.82 + 0.25 = 1.07$$

   New observations scatter around $f(2)$ with variance $\sigma^2$. No model can predict that part, so 0.25 is a floor under both errors.

8. 8Averaging cuts variance

   $$\bar f = \tfrac15 \textstyle\sum \hat f_i = 4.0, \qquad \text{Var} \approx \frac{0.82}{5} = 0.164$$

   If the five complex models' errors are independent, averaging them keeps bias at 0 and divides variance by 5. That is the idea behind bagging and random forests.

The rule in plain words

As model complexity grows, bias falls (it can fit more shapes) and variance rises (it fits the noise of each sample).

Pick the complexity where the error on held-out data is lowest. Regularization and more training data reduce variance without making the model simpler.

The code

Read this code after you have computed the five-model example by hand. It will make more sense.

void bias_variance(double preds[], int k, double truth, double out[]) {
    double mean = 0, variance = 0;
    for (int i = 0; i < k; i++) mean += preds[i] / k;      /* the "average model" */
    for (int i = 0; i < k; i++) variance += (preds[i] - mean) * (preds[i] - mean) / k;
    out[0] = (mean - truth) * (mean - truth);              /* bias squared */
    out[1] = variance;                                     /* spread around the average */
}

flowchart LR
  A["simple model"] -->|"more flexibility"| B["balanced model"]
  B -->|"more flexibility"| C["complex model"]
  A -.- A1["high bias, low variance: underfit"]
  B -.- B1["lowest test error"]
  C -.- C1["low bias, high variance: overfit"]
          

Putting numbers on it

For squared error, the expected error of the learned model at a point splits exactly into three parts:

Simple models above: $1 + 0.02 = 1.02$ plus noise. Complex models: $0 + 0.82 = 0.82$ plus noise. Here the complex model wins slightly; with less data its variance would grow.

Common mistakes

• Fixing overfitting by adding more model capacity.
• Thinking more data reduces bias; it mainly reduces variance.
• Forgetting the irreducible noise term: even the perfect model has error $\sigma^2$.

Try it yourself

Did you know?

Key points to remember

If you remember one thing from each line, remember the because.

• Simple models underfit because they cannot represent the true shape, which is bias.
• Complex models overfit because they also fit the noise of their particular training sample, which is variance.
• There is an optimal complexity because bias falls and variance rises as flexibility grows.
• More data mainly helps variance because noise averages out over more examples.
• Some error is unavoidable because the noise $\sigma^2$ is in the data itself.

Self-check questions

Answer these in order, out loud. They follow the page, so your answers become the script of your video.

1. How are bias and variance like two archers?
   Sample answer

   High bias is an archer who always hits the same wrong spot; high variance is one whose arrows are centred but scattered everywhere.

2. Compare the flat line, the smooth U and the wiggly curve on the 7 points.
   Sample answer

   The line underfits (high training and test error), the wiggly curve overfits (zero training error, large test error), the U curve has low error on both.

3. Why does the code average the predictions before computing the variance?

   A good answer says what variance is measured around.

   Sample answer

   Variance is the spread of the predictions around their own average, so the average has to be known first; the same average gives the bias.

4. So what are bias and variance, in one sentence each?
   Sample answer

   Bias is how far the average model is from the truth; variance is how much the model changes from one training sample to another.

5. Why does making a model more flexible lower bias but raise variance?

   A good answer mentions noise.

   Sample answer

   More flexibility lets it follow the true shape (less bias), but also the random noise of each sample, so different samples give very different models (more variance).

6. What three terms make up the expected squared error?
   Sample answer

   Bias², variance, and the irreducible noise $\sigma^2$.

7. Training error 2%, test error 25%: what is wrong and what do you do?
   Sample answer

   Overfitting (high variance). Add data, regularize, simplify the model or stop training earlier.

8. If you had to teach this to a friend in 2 minutes, what would you leave out, and why?
   Sample answer

   I'd leave out the decomposition formula and double descent. I'd keep the archer analogy, the three fits on the U data, and "too simple = bias, too flexible = variance, pick the middle with test data".

External resources

• scikit-learn: Underfitting vs overfitting: polynomial fits of degree 1, 4 and 15 with code.
• Wikipedia: Bias–variance trade-off: the decomposition and its derivation.
• Google ML Crash Course: Overfitting: generalisation with examples.

Videos

• YouTube search: StatQuest bias and variance: StatQuest's short, clear explanation.