Data preprocessing

Prerequisites

• Mean and standard deviation.
• Numeric versus categorical features.
• Training versus test data.

The analogy

How it works

Golden rule: learn every preprocessing parameter (means, standard deviations, categories) from the training data only, then apply the same parameters to validation and test data.

Start with one column by hand

Ages $[20, 30, 40, 50]$. Predict, then open "Check".

Finish the picture: categories and gaps

Now handle a categorical column and a missing value.

• Colours red, green, blue, green. Predict the one-hot columns.
  Check

  Three columns (is_red, is_green, is_blue): rows [1, 0, 0], [0, 1, 0], [0, 0, 1], [0, 1, 0].

• Ages $[20, ?, 40, 50]$. Predict the mean imputation, and one risk.
  Check

  Fill with $(20 + 40 + 50)/3 \approx 36.7$. Risk: it shrinks the spread and hides that the value was missing; adding an "age_missing" flag column helps.

Full worked calculation

The page's age column $[20, 30, 40, 50]$, preprocessed three ways, then a new test age of 60 run through the same steps. Last: what mean imputation does to the spread of $[20, ?, 40, 50]$.

1. 1Mean

   $$\mu = \frac{20 + 30 + 40 + 50}{4} = 35$$

   The centre of the column. Subtracting it makes "average age" become 0.

2. 2Standard deviation

   $$\sigma = \sqrt{\frac{225 + 25 + 25 + 225}{4}} = \sqrt{125} = 11.18$$

   The typical distance from the mean. Dividing by it makes the unit "one standard deviation" instead of "one year", so ages and incomes become comparable.

3. 3Standardize each value

   $$z = \frac{x - 35}{11.18}: \quad 20 \mapsto -1.342, \quad 30 \mapsto -0.447, \quad 40 \mapsto 0.447, \quad 50 \mapsto 1.342$$

   The new column has mean 0 and standard deviation 1. Gradient descent, k-NN and SVMs all behave better when every feature is on this scale.

4. 4Min-max scaling

   $$x' = \frac{x - 20}{50 - 20}: \quad [0,\; 0.333,\; 0.667,\; 1]$$

   An alternative that squeezes the training values into $[0, 1]$. Useful for pixel values or when a bounded range matters.

5. 5A test value, with training statistics

   $$z_{60} = \frac{60 - 35}{11.18} = 2.236, \qquad x'_{60} = \frac{60 - 20}{30} = 1.333$$

   The test row uses $\mu$, $\sigma$, min and max from training, unchanged. Min-max output above 1 is normal for unseen data: the range is only guaranteed on the training set.

6. 6One-hot encoding

   $$\text{red} \mapsto [1, 0, 0], \quad \text{green} \mapsto [0, 1, 0], \quad \text{blue} \mapsto [0, 0, 1]$$

   Coding colours as 1, 2, 3 would tell the model "blue is three times red". Separate 0/1 columns carry no fake order.

7. 7Mean imputation shrinks the spread

   $$[20, ?, 40, 50]: \; \mu = 36.67, \;\; \sigma_{\text{observed}} = 12.47 \;\longrightarrow\; [20, 36.67, 40, 50]: \; \sigma = 10.80$$

   The filled value sits exactly on the mean, so it adds a row with zero deviation. The column now looks 13% less spread out than it really is. Adding an "age_missing" column keeps the information that a value was invented.

The rule in plain words

Scale numeric features (standardize, or min-max). Encode categories (one-hot for unordered, ordinal codes only for truly ordered ones). Impute missing values (mean, median or most frequent, plus a missing flag).

Fit all of these on the training data, store the parameters, and apply exactly the same transformation to every later dataset.

The code

Read this code after you have standardized the ages by hand. It will make more sense.

void standardize(double x[], int n, double stats[]) {     /* z = (x - mean) / sd */
    double mean = 0, variance = 0;
    for (int i = 0; i < n; i++) mean += x[i] / n;
    for (int i = 0; i < n; i++) variance += (x[i] - mean) * (x[i] - mean) / n;
    double sd = sqrt(variance);
    for (int i = 0; i < n; i++) x[i] = (x[i] - mean) / sd;
    stats[0] = mean; stats[1] = sd;           /* keep them to transform test data */
}

flowchart TD
  A{"What kind of column?"} -->|numeric| B{"Missing values?"}
  B -->|yes| C["impute (median or mean) + missing flag"]
  B -->|no| D["scale: standardize or min-max"]
  C --> D
  A -->|"categorical, no order"| E["one-hot encode"]
  A -->|"categorical, ordered (S < M < L)"| F["ordinal codes 0, 1, 2"]
          

Putting numbers on it

The two common scalings, with our ages:

Common mistakes

• Computing the mean and sd on the full dataset before splitting (data leakage).
• Label-encoding unordered categories as 1, 2, 3, which invents an order.
• Min-max scaling with data containing extreme outliers, which squashes all normal values together.

Try it yourself

Did you know?

Key points to remember

If you remember one thing from each line, remember the because.

• Scaling matters because distance and gradient methods treat a big-number feature as more important.
• Standardization gives mean 0 and sd 1 because it subtracts the mean and divides by the sd.
• One-hot encoding avoids fake order because each category gets its own independent column.
• Parameters come from training data only because using test data would leak information.
• A missing flag helps because the fact that a value was missing can itself be informative.

Self-check questions

Answer these in order, out loud. They follow the page, so your answers become the script of your video.

1. How is preprocessing like converting units in a recipe?
   Sample answer

   Before comparing "2 cups" and "300 grams" you convert to one unit; preprocessing puts features on comparable scales and turns words into numbers.

2. Standardize the ages [20, 30, 40, 50].
   Sample answer

   Mean 35, sd about 11.18, so z ≈ [−1.34, −0.45, 0.45, 1.34].

3. Why does standardize store the mean and sd in stats?

   A good answer mentions test data.

   Sample answer

   The same mean and sd, learned from training data, must be reused to transform test data and new data in production.

4. So what is preprocessing, in one sentence?
   Sample answer

   Transforming raw data into a model-ready form by scaling numbers, encoding categories and filling missing values.

5. Why does KNN need scaling but a decision tree doesn't?

   A good answer compares distances with splits.

   Sample answer

   KNN measures distances, so a feature with large numbers dominates. A tree splits one feature at a time on thresholds, which is unaffected by the feature's scale.

6. What are the formulas for standardization and min-max scaling?
   Sample answer

   $(x - \mu)/\sigma$ and $(x - \min)/(\max - \min)$.

7. How should a categorical colour column be encoded?
   Sample answer

   One-hot: one 0/1 column per colour, because the colours have no natural order.

8. If you had to teach this to a friend in 2 minutes, what would you leave out, and why?
   Sample answer

   I'd leave out log transforms and missing flags. I'd keep the recipe-units analogy, the four-age standardization, one-hot colours, and "fit on training data, apply everywhere".

External resources

• scikit-learn: Preprocessing data: scalers, encoders and imputers with code.
• Google ML Crash Course: Normalization: scaling numerical data, visually.
• Wikipedia: Feature scaling: methods and motivation.

Videos

• YouTube search: feature scaling explained: standardization versus normalization.