Data leakage

Prerequisites

• Preprocessing and standardization.
• Train / validation / test splits.
• The idea that a test set simulates the future.

The analogy

How it works

Test for leakage: for every feature and every preprocessing step, ask: "would I know this, computed this way, at the moment of making a real prediction?" If not, it leaks.

Start with a leaky scaler by hand

Training ages $[20, 30, 40]$, test age $80$. Predict, then open "Check".

Finish the picture: target leakage

A hospital model predicts pneumonia. One feature is "received antibiotics", recorded during the hospital stay.

• Validation accuracy is 98%. Predict why it is so high.
  Check

  Patients with pneumonia get antibiotics, so the feature almost reveals the label. The model learned "antibiotics → pneumonia".

• Deployed at admission time. Predict what happens.
  Check

  At admission no antibiotics have been given yet, so the feature is always "no" and accuracy collapses. Fix: only use information available at prediction time.

Full worked calculation

The page's scaler, done both ways in full. Training ages $[20, 30, 40]$, one test person aged 80. A simple anomaly rule then flags anyone with $|z| > 3$: we see which version of the pipeline catches the unusual person.

1. 1Fit the scaler on training rows only

   $$\mu = \frac{20 + 30 + 40}{3} = 30, \qquad \sigma^2 = \frac{(-10)^2 + 0^2 + 10^2}{3} = 66.67, \qquad \sigma = 8.165$$

   These two numbers are learned parameters, just like model weights. They may only come from data the model is allowed to see during training.

2. 2Transform the test row

   $$z_{80} = \frac{80 - 30}{8.165} = 6.124$$

   The test person is measured with the training ruler. 6 standard deviations away: very unusual, which is the truth.

3. 3The leaky fit, using all rows

   $$\mu = \frac{20 + 30 + 40 + 80}{4} = 42.5, \qquad \sigma^2 = \frac{506.25 + 156.25 + 6.25 + 1406.25}{4} = 518.75$$

   The squared deviations are $(-22.5)^2, (-12.5)^2, (-2.5)^2, 37.5^2$. The test value alone contributes 1406 of the 2075: it dominates the scale.

4. 4Transform with leaked statistics

   $$\sigma = \sqrt{518.75} = 22.776, \qquad z_{80} = \frac{80 - 42.5}{22.776} = 1.646$$

   The 80-year-old pulled the mean towards themself and stretched the spread, so they now look fairly ordinary.

5. 5Apply the rule |z| > 3

   $$\text{correct: } 6.124 > 3 \Rightarrow \text{flagged}, \qquad \text{leaky: } 1.646 < 3 \Rightarrow \text{missed}$$

   Same data, same rule, opposite decisions. Even the training rows shifted: 30 became $-0.549$ instead of 0, so the model trains on different numbers.

6. 6Why the evaluation lies

   $$\text{evaluation used } \mu, \sigma \text{ that already knew the test set}$$

   In production a new person arrives one at a time and never influences the scaler. Any score measured with the leaky pipeline describes a situation that will never happen.

7. 7The fix inside cross-validation

   $$\text{for each fold: } \texttt{fit\_scaler}(\text{train fold}) \;\to\; \texttt{transform}(\text{train fold}),\; \texttt{transform}(\text{test fold})$$

   Refit the scaler (and any imputer, encoder or feature selector) inside every fold, on that fold's training rows only. Libraries do this automatically when the steps are wrapped in a pipeline.

The rule in plain words

Split the data first, and fit every learned step (scaling, imputation, feature selection, encoding) on the training part only, ideally inside a pipeline.

Drop any feature that would not be known at prediction time, remove duplicates across splits, and split time-ordered data by time.

The code

Read this code after you have compared the two scalers by hand. It will make more sense.

void fit_scaler(double train[], int n, double stats[]) {   /* learn from training rows only */
    double mean = 0, variance = 0;
    for (int i = 0; i < n; i++) mean += train[i] / n;
    for (int i = 0; i < n; i++) variance += (train[i] - mean) * (train[i] - mean) / n;
    stats[0] = mean; stats[1] = sqrt(variance);
}

void transform(double x[], int n, double stats[]) {        /* apply to any rows, incl. test */
    for (int i = 0; i < n; i++) x[i] = (x[i] - stats[0]) / stats[1];
}

flowchart TD
  A["a feature or preprocessing step"] --> B{"computed using test or future rows?"}
  B -->|yes| L1["leak: fit it on training rows only"]
  B -->|no| C{"only known after the outcome?"}
  C -->|yes| L2["target leak: drop the feature"]
  C -->|no| D{"same rows in train and test?"}
  D -->|yes| L3["leak: remove duplicates / group the split"]
  D -->|no| OK["safe"]
          

Putting numbers on it

The z-score of the test age under the two scalers:

The leaky scaler made an extreme case look ordinary. In general, leakage inflates evaluation scores, sometimes from mediocre to near-perfect.

Common mistakes

• Imputing missing values or selecting features using the whole dataset.
• Using features that are recorded after the event being predicted.
• Splitting rows randomly when the same person, patient or day appears many times.

Try it yourself

Did you know?

Key points to remember

If you remember one thing from each line, remember the because.

• Leakage inflates scores because the model uses information it won't have in real use.
• Split before preprocessing because fitted parameters must come from training data only.
• Future-only features leak because they partly encode the answer.
• Duplicates leak because the model can memorise them instead of generalising.
• Time series must be split by time because a random split lets the model train on the future.

Self-check questions

Answer these in order, out loud. They follow the page, so your answers become the script of your video.

1. How is data leakage like seeing the exam answers the night before?
   Sample answer

   The score looks perfect but says nothing about real ability, because the answers were available during preparation.

2. Scale the test age 80 correctly and leakily, with training ages [20, 30, 40].
   Sample answer

   Correct: mean 30, sd 8.16, z ≈ 6.1. Leaky: mean 42.5, sd 22.8, z ≈ 1.6.

3. Why is fit_scaler separate from transform?

   A good answer says which rows each function sees.

   Sample answer

   Fitting must only see training rows, while transforming must be applied to training, test and new rows with the same parameters; separating them makes that order explicit.

4. So what is data leakage, in one sentence?
   Sample answer

   Using, during training or evaluation, information that would not be available when the model makes real predictions.

5. Why does the pneumonia model fail after deployment?

   A good answer mentions timing.

   Sample answer

   "Received antibiotics" is only known after diagnosis. At admission it is always "no", so the shortcut the model learned disappears.

6. Name four kinds of leakage.
   Sample answer

   Preprocessing on all data, target leakage from future-only features, duplicates across splits, and random splits of time-ordered data (plus tuning on the test set).

7. How do pipelines help prevent leakage?
   Sample answer

   They bundle preprocessing and model so that, in each CV fold, everything is fitted on that fold's training rows only.

8. If you had to teach this to a friend in 2 minutes, what would you leave out, and why?
   Sample answer

   I'd leave out the time-series and duplicate cases. I'd keep the exam-answers analogy, the leaky scaler numbers, the antibiotics example, and "would I know this at prediction time?".

External resources

• scikit-learn: Common pitfalls: data leakage and how pipelines prevent it.
• Wikipedia: Leakage (machine learning): types and examples.
• Kaggle Learn: Intermediate ML: includes a hands-on lesson on leakage.

Videos

• YouTube search: data leakage explained: pick a short explainer.