Cross-validation

Prerequisites

• Overfitting and why test data must be unseen.
• Mean and standard deviation.
• A metric to score each fold.

The analogy

How it works

Rule: each fold's test block must be completely unseen during that fold's training, including any preprocessing fitted on the data.

Start with a 5-fold split on 100 samples

Samples are numbered 1–100 (shuffled beforehand). Fold $f$ tests on block $f$ and trains on the other four. Predict, then open "Check".

Finish the picture: choosing with CV

Two settings of a model give these 5-fold mean scores: setting A 0.80 ± 0.03, setting B 0.81 ± 0.08.

• Is B clearly better? Predict.
  Check

  No. The difference (0.01) is far smaller than B's spread (0.08); the two are statistically hard to tell apart, and A is more stable.

• After choosing, what do you train? Predict.
  Check

  One final model on all 100 samples with the chosen setting. CV was only for choosing; a separate untouched test set gives the final honest score.

Full worked calculation

The page's 5-fold cross-validation on $n = 100$ shuffled samples, with fold scores 0.82, 0.78, 0.85, 0.80, 0.75. We compute the fold boundaries with the code's formula, the mean and spread, and then use them to compare two settings.

1. 1Fold boundaries

   $$\text{start} = \left\lfloor \frac{f \cdot n}{k} \right\rfloor, \quad \text{end} = \left\lfloor \frac{(f + 1)\, n}{k} \right\rfloor \;\Rightarrow\; f = 2:\; \left[\tfrac{200}{5}, \tfrac{300}{5}\right) = [40, 60)$$

   With 0-based indices, fold 3 ($f = 2$ in code) tests on indices 40–59, which are samples 41–60. The other 80 samples train the model for that fold.

2. 2When n does not divide evenly

   $$n = 10,\; k = 3: \quad [0, 3),\; [3, 6),\; [6, 10) \quad \text{sizes } 3, 3, 4$$

   Integer division spreads the leftover sample so block sizes differ by at most 1. Every sample still lands in exactly one test block.

3. 3Mean score

   $$\bar s = \frac{0.82 + 0.78 + 0.85 + 0.80 + 0.75}{5} = \frac{4.00}{5} = 0.80$$

   Each of the 100 samples was scored exactly once by a model that never saw it, so 0.80 uses all the data for testing without cheating.

4. 4Spread across folds

   $$\text{sd} = \sqrt{\frac{0.0058}{5}} = \sqrt{0.00116} \approx 0.034$$

   Squared deviations from the table, averaged, then square-rooted. Dividing by $k - 1 = 4$ instead gives 0.038; either way, report "0.80 ± 0.03".

5. 5What one split would have told you

   $$\text{single 80/20 split: anywhere from } 0.75 \text{ to } 0.85$$

   A plain hold-out set is just one of these folds. Depending on luck, you would have reported 0.75 or 0.85: a gap of 0.10 that has nothing to do with the model.

6. 6Compare two settings

   $$A: 0.80 \pm 0.03, \quad B: 0.81 \pm 0.08 \;\Rightarrow\; 0.81 - 0.80 = 0.01 \ll 0.08$$

   The difference is eight times smaller than B's fold-to-fold noise. Prefer A: the same score, and far more stable.

7. 7The cost

   $$k = 5: \; 5 \text{ trainings on } 80 \text{ samples}, \qquad k = 100 \text{ (leave-one-out)}: \; 100 \text{ trainings on } 99$$

   More folds mean bigger training sets (less pessimistic scores) but more compute. 5 or 10 is the usual compromise.

The rule in plain words

Shuffle, then split the data into $k$ equal folds. For each fold, train on the other $k - 1$ folds and score on this one.

Report the mean and spread of the $k$ scores. Use it to compare models or tune settings, never touching the final test set.

The code

Read this code after you have laid out the folds by hand. It will make more sense.

void fold_range(int n, int k, int f, int range[]) {   /* test indices [range[0], range[1]) */
    range[0] = (f * n) / k;
    range[1] = ((f + 1) * n) / k;                     /* sizes differ by at most 1 */
}

double mean_score(double s[], int k) {               /* average of the k fold scores */
    double m = 0;
    for (int i = 0; i < k; i++) m += s[i] / k;
    return m;
}

flowchart TD
  A["shuffle and split data into k folds"] --> B["fold i = test, others = train"]
  B --> C["fit the model (and preprocessing) on train only"]
  C --> D["score on fold i"]
  D --> E{"all k folds done?"}
  E -->|no| B
  E -->|yes| F["report mean ± spread of the k scores"]
          

Putting numbers on it

The cross-validation score is the mean of the fold scores, reported with their standard deviation:

The cost is $k$ trainings, so 10-fold CV is about 10 times slower than one split.

Common mistakes

• Reporting only the mean, without the spread across folds.
• Using plain random folds on time-series data, so the model trains on the future.
• Tuning on the test set after cross-validation, which makes the final score optimistic.

Try it yourself

Did you know?

Key points to remember

If you remember one thing from each line, remember the because.

• CV gives a more reliable score because it averages over several different test blocks.
• Every sample is tested once because each fold serves as the test block exactly once.
• The spread matters because a small difference in mean can be pure noise.
• Preprocessing goes inside the folds because fitting it on all data leaks test information.
• CV costs $k$ trainings because a fresh model is trained for every fold.

Self-check questions

Answer these in order, out loud. They follow the page, so your answers become the script of your video.

1. How is cross-validation like rotating exam sets?
   Sample answer

   You study 4 sets and are tested on the fifth, rotating until every set has been the exam; the average mark is more trustworthy than one exam.

2. Describe 5-fold CV on 100 samples and compute the mean of 0.82, 0.78, 0.85, 0.80, 0.75.
   Sample answer

   Five blocks of 20; each fold trains on 80 and tests on 20. The mean score is 0.80.

3. Why does fold_range compute (f * n) / k with the multiplication first?

   A good answer mentions integer division.

   Sample answer

   In integer arithmetic, dividing first would round $n / k$ down and could skip samples. Multiplying first makes the blocks cover all $n$ samples, with sizes differing by at most one.

4. So what is $k$-fold cross-validation, in one sentence?
   Sample answer

   Splitting the data into $k$ folds and training $k$ times, each time testing on a different fold, then averaging the scores.

5. Why is a single train/test split less reliable?

   A good answer mentions luck.

   Sample answer

   Its score depends on which samples happened to land in the test set; CV averages over $k$ different test sets and shows the spread.

6. How many models does $k$-fold CV train, and what is leave-one-out?
   Sample answer

   $k$ models. Leave-one-out is $k = n$: each sample is a test set on its own.

7. How do you avoid leakage inside cross-validation?
   Sample answer

   Fit every preprocessing step on the training part of each fold only, then apply it to that fold's test part, ideally with a pipeline.

8. If you had to teach this to a friend in 2 minutes, what would you leave out, and why?
   Sample answer

   I'd leave out stratified and time-series variants. I'd keep the rotating-exam analogy, the 5 × 20 split with its scores, and "every sample tested once, report mean ± spread".

External resources

• scikit-learn: Cross-validation: all CV strategies with diagrams and code.
• Wikipedia: Cross-validation: types and theory.
• Google ML Crash Course: Dividing datasets: training, validation and test sets.

Videos

• YouTube search: StatQuest cross-validation: StatQuest's short explainer.