Regression metrics

Prerequisites

• Regression: predicting a number.
• Mean, absolute value and squares.
• Square roots.

The analogy

How it works

Notation: $y_i$ is the true value, $\hat y_i$ the prediction, $e_i = y_i - \hat y_i$ the error (residual), and $\bar y$ the mean of the true values.

Start with four predictions by hand

True values $y = [3, 5, 2, 7]$, predictions $\hat y = [2.5, 5, 4, 8]$. Predict, then open "Check".

Finish the picture: one outlier

Change the last prediction from 8 to 17 (error −10).

• MAE. Predict the new value.
  Check

  $(0.5 + 0 + 2 + 10)/4 = 3.125$: about 3.6 times larger.

• MSE. Predict it, and compare the growth.
  Check

  $(0.25 + 0 + 4 + 100)/4 \approx 26.1$: about 20 times larger. MSE is dominated by the single big miss.

Full worked calculation

The page's four houses: true prices $y = [3, 5, 2, 7]$, predictions $\hat y = [2.5, 5, 4, 8]$ (hundreds of thousands). Every column of the computation is in the table; then we compare with the "always predict the mean" baseline and see what one outlier does to each metric.

1. 1Errors (residuals)

   $$e_i = y_i - \hat y_i = [0.5,\; 0,\; -2,\; -1]$$

   Positive means the model guessed too low, negative too high. The signs cancel if you just add them ($-2.5$), which is why every metric first removes the sign.

2. 2MAE

   $$\text{MAE} = \frac{0.5 + 0 + 2 + 1}{4} = \frac{3.5}{4} = 0.875$$

   On average the prediction is off by 87,500. Every unit of error counts the same, whatever the size of the miss.

3. 3MSE and RMSE

   $$\text{MSE} = \frac{0.25 + 0 + 4 + 1}{4} = 1.3125, \qquad \text{RMSE} = \sqrt{1.3125} = 1.146$$

   Squaring makes the miss of 2 contribute 4 of the 5.25 total. RMSE puts the result back in the original unit (about 115,000), and is always at least as large as MAE.

4. 4Total variation around the mean

   $$\bar y = \frac{17}{4} = 4.25, \qquad \sum (y_i - \bar y)^2 = 14.75$$

   This is the squared error of the laziest possible model, one that always predicts 4.25. Its MSE would be $14.75 / 4 = 3.69$.

5. 5R²

   $$R^2 = 1 - \frac{\sum e_i^2}{\sum (y_i - \bar y)^2} = 1 - \frac{5.25}{14.75} = 1 - 0.356 = 0.644$$

   Our model leaves 35.6% of the baseline's squared error, so it explains 64.4% of the variation. $R^2 = 0$ means "no better than the mean", 1 means perfect.

6. 6One outlier: prediction 8 → 17

   $$\text{MAE} = \frac{3.5 + 9}{4} = 3.125, \qquad \text{MSE} = \frac{5.25 - 1 + 100}{4} = 26.06$$

   The fourth error grows from $-1$ to $-10$. MAE grows 3.6 times; MSE grows 20 times, because the 10 is squared to 100.

7. 7R² can go negative

   $$R^2 = 1 - \frac{104.25}{14.75} = 1 - 7.07 = -6.07$$

   The model is now seven times worse than always predicting the mean. A negative $R^2$ is not a bug: it means the model is worse than the baseline.

The rule in plain words

Compute each error $e_i = y_i - \hat y_i$. MAE averages $|e_i|$; MSE averages $e_i^2$; RMSE is $\sqrt{\text{MSE}}$, back in the original units.

R² $= 1 - \sum e_i^2 / \sum (y_i - \bar y)^2$ compares the model with predicting the mean.

The code

Read this code after you have computed the metrics by hand. It will make more sense.

void reg_metrics(double y[], double p[], int n, double out[]) {   /* MAE, MSE, R^2 */
    double mean = 0, abs_sum = 0, sq_sum = 0, tot = 0;
    for (int i = 0; i < n; i++) mean += y[i] / n;
    for (int i = 0; i < n; i++) {
        double e = y[i] - p[i];                 /* error (residual) */
        abs_sum += fabs(e);
        sq_sum += e * e;
        tot += (y[i] - mean) * (y[i] - mean);   /* variation around the mean */
    }
    out[0] = abs_sum / n;                       /* MAE */
    out[1] = sq_sum / n;                        /* MSE */
    out[2] = 1 - sq_sum / tot;                  /* R^2 */
}

flowchart TD
  A{"What do you care about?"} -->|"typical error size, robust to outliers"| MAE["MAE"]
  A -->|"big errors are very costly"| MSE["MSE / RMSE"]
  A -->|"error in the original units"| RMSE["RMSE or MAE"]
  A -->|"how much variation is explained"| R2["R squared"]
          

Putting numbers on it

The definitions, with our four houses:

Common mistakes

• Forgetting the absolute value in MAE, so positive and negative errors cancel out.
• Comparing MSE values across datasets with different scales.
• Reading $R^2 = 0.64$ as "64% of predictions are correct"; it's the share of variance explained.

Try it yourself

Did you know?

Key points to remember

If you remember one thing from each line, remember the because.

• MAE treats all errors equally because it averages their absolute sizes.
• MSE punishes big errors because squaring makes large errors grow much faster than small ones.
• RMSE is easier to interpret because the square root brings it back to the units of $y$.
• $R^2 = 0$ means "as good as the mean" because the mean predictor's squared error equals the total variation.
• $R^2$ can be negative because a model can make larger errors than the mean predictor.

Self-check questions

Answer these in order, out loud. They follow the page, so your answers become the script of your video.

1. How are MAE and MSE like scoring darts?
   Sample answer

   MAE is the average distance from the bullseye; MSE squares distances, so one wild throw counts hugely.

2. Compute MAE, MSE and $R^2$ for truth [3, 5, 2, 7] and predictions [2.5, 5, 4, 8].
   Sample answer

   MAE 0.875, MSE about 1.31, $R^2$ about 0.64.

3. Why does the code compute the mean in a first loop before the main loop?

   A good answer mentions the $R^2$ denominator.

   Sample answer

   The total variation $\sum (y_i - \bar y)^2$ needs the mean, so it must be known before the second loop adds up those squares.

4. So what do MAE, MSE and $R^2$ measure, in one sentence each?
   Sample answer

   MAE: average absolute error. MSE: average squared error, sensitive to big misses. $R^2$: share of the variation in $y$ explained by the model.

5. Why did one outlier multiply the MSE by about 20 but the MAE by about 3.6?

   A good answer compares 10 and 10².

   Sample answer

   MAE adds the error 10 as it is, while MSE adds $10^2 = 100$, so the outlier dominates the squared sum.

6. Which metrics are in the same units as $y$?
   Sample answer

   MAE and RMSE; MSE is in squared units and $R^2$ has no units.

7. When can $R^2$ be negative?
   Sample answer

   When the model's squared errors exceed the total variation, i.e. it predicts worse than always guessing the mean, which can happen on test data.

8. If you had to teach this to a friend in 2 minutes, what would you leave out, and why?
   Sample answer

   I'd leave out RMSE and the negative-$R^2$ case. I'd keep the darts analogy, the four-house example, the outlier effect, and "MAE robust, MSE punishes big misses, $R^2$ = variance explained".

External resources

• scikit-learn: Regression metrics: MAE, MSE, $R^2$ and more with code.
• Wikipedia: Coefficient of determination: what $R^2$ means and its pitfalls.
• Google ML Crash Course: Loss: squared versus absolute loss.

Videos

• YouTube search: StatQuest R-squared: StatQuest explains $R^2$ clearly.