Principal component analysis (PCA)

Prerequisites

• Variance and covariance.
• Vectors, dot products and projections.
• Standardization.

The analogy

How it works

Before PCA: centre each feature (subtract its mean) and usually standardize it, so that features measured in large units don't dominate the variance.

Start with explained variance by hand

The covariance matrix of 4 standardized-ish features has eigenvalues $\lambda = 4, 2, 1, 0.5$. Each eigenvalue is the variance along one principal component. Predict, then open "Check".

Finish the picture: projecting a point

Take 2-D points (1, 1), (2, 2), (3, 3). They all lie on one line.

• The first component. Predict its direction.
  Check

  $v_1 = (1, 1)/\sqrt{2} \approx (0.707, 0.707)$: along the line.

• Project the centred point $(3, 3) - (2, 2) = (1, 1)$ onto $v_1$. Predict the 1-D coordinate.
  Check

  $1 \cdot 0.707 + 1 \cdot 0.707 \approx 1.414$. One number per point, with no information lost here, since PC2 has variance 0.

Full worked calculation

Full PCA by hand on four 2-D points, (1, 1), (2, 3), (3, 2), (4, 4): centre, covariance, eigenvalues, eigenvector, projection, and what is lost when we keep only one component.

1. 1Centre the data

   $$\bar x = \frac{1 + 2 + 3 + 4}{4} = 2.5, \qquad \bar y = \frac{1 + 3 + 2 + 4}{4} = 2.5$$

   PCA looks for directions of spread around the centre, so we subtract the mean from every point first. Without this step the first "component" would just point at the mean.

2. 2Covariance matrix

   $$C = \frac{1}{n - 1}\begin{pmatrix} \sum x_c^2 & \sum x_c y_c \\ \sum x_c y_c & \sum y_c^2 \end{pmatrix} = \frac{1}{3}\begin{pmatrix} 5 & 4 \\ 4 & 5 \end{pmatrix} = \begin{pmatrix} 1.667 & 1.333 \\ 1.333 & 1.667 \end{pmatrix}$$

   The diagonal is the variance of each feature; the off-diagonal says they move together (positive). The sums come straight from the table.

3. 3Eigenvalues

   $$\det(C - \lambda I) = (1.667 - \lambda)^2 - 1.333^2 = 0 \;\Rightarrow\; \lambda = 1.667 \pm 1.333 = 3 \text{ and } 0.333$$

   Each eigenvalue is the variance of the data along one principal direction. For a symmetric 2×2 matrix with equal diagonals, the eigenvalues are "diagonal ± off-diagonal".

4. 4Eigenvectors

   $$\lambda_1 = 3: \; v_1 = \tfrac{1}{\sqrt 2}(1, 1) \approx (0.707, 0.707), \qquad \lambda_2 = 0.333: \; v_2 = \tfrac{1}{\sqrt 2}(1, -1)$$

   Check: $C(1, 1)^\top = (3, 3)^\top = 3 \cdot (1, 1)^\top$. PC1 runs along the diagonal, where the points are spread out; PC2 is perpendicular to it.

5. 5Explained variance

   $$\frac{\lambda_1}{\lambda_1 + \lambda_2} = \frac{3}{3.333} = 0.9$$

   PC1 alone keeps 90% of the total variance. If 90% is enough for the task, we can describe each point with one number instead of two.

6. 6Project onto PC1

   $$z = x_c \cdot v_1: \quad (-1.5, -1.5) \cdot (0.707, 0.707) = -2.121, \qquad (-0.5, 0.5) \cdot (0.707, 0.707) = 0$$

   One dot product per point gives its new 1-D coordinate. The variance of these coordinates, $(4.5 + 0 + 0 + 4.5)/3 = 3$, is exactly $\lambda_1$.

7. 7What is lost

   $$\hat x = \bar x + z\,v_1: \quad (2, 3) \to (2.5, 2.5), \quad \text{error} = 0.5^2 + 0.5^2 = 0.5$$

   Reconstructing from one coordinate puts (2, 3) and (3, 2) both at the centre. The total squared error is $0 + 0.5 + 0.5 + 0 = 1 = \lambda_2 (n - 1)$: we lose exactly the variance of the dropped component.

The rule in plain words

Centre (and usually standardize) the data, compute its covariance matrix, and find its eigenvectors (directions) and eigenvalues (variances), sorted largest first.

Keep the top $k$ components that explain enough variance (e.g. 90–95%), and represent each point by its projections onto them.

The code

Read this code after you have computed the shares by hand. It will make more sense.

double explained(double eig[], int m, int k) {        /* share of variance in the first k */
    double top = 0, total = 0;                       /* eig sorted from largest */
    for (int i = 0; i < m; i++) { total += eig[i]; if (i < k) top += eig[i]; }
    return top / total;
}

double project(double x[], double v[], int d) {      /* coordinate of x along unit vector v */
    double s = 0;
    for (int j = 0; j < d; j++) s += x[j] * v[j];
    return s;
}

flowchart LR
  A["data, d features"] --> B["centre and standardize"]
  B --> C["covariance matrix"]
  C --> D["eigenvectors + eigenvalues, sorted"]
  D --> E["keep top k by explained variance"]
  E --> F["project: data with k features"]
          

How costly is it?

The explained-variance ratio of the first $k$ components, with eigenvalues sorted from largest:

Best for: compressing many correlated numeric features, speeding up later models, and visualising high-dimensional data in 2-D or 3-D.

• Pro: Removes redundancy; components are uncorrelated.
• Pro: Fast, deterministic, with a clear measure of information kept (explained variance).
• Con: Only finds linear structure.
• Con: Components are mixtures of features, so they are harder to interpret; sensitive to scaling.

Common mistakes

• Running PCA on unscaled features, so the biggest-unit feature becomes PC1.
• Fitting PCA on all data before the train/test split (leakage).
• Interpreting components as if they were original features.

Try it yourself

Did you know?

Key points to remember

If you remember one thing from each line, remember the because.

• Eigenvalues measure variance per component because they are the variance of the data projected onto each eigenvector.
• The first components keep the most information because they are sorted by variance.
• Components are uncorrelated because they are orthogonal eigenvectors of the covariance matrix.
• Scaling matters because variance depends on the units of each feature.
• PCA can drop useful signal because it ignores the target entirely.

Self-check questions

Answer these in order, out loud. They follow the page, so your answers become the script of your video.

1. How is PCA like choosing the best camera angle for a surfboard?
   Sample answer

   It picks the views that show the most spread: the first component is the best single view, the next the best one at a right angle.

2. With eigenvalues 4, 2, 1, 0.5, how much variance do the first two components explain?
   Sample answer

   $6/7.5 = 80\%$.

3. Why does explained need the eigenvalues sorted from largest?

   A good answer says what "the first $k$" must mean.

   Sample answer

   It adds the first $k$ entries, which must be the $k$ largest variances; unsorted input would mix in small components.

4. So what is PCA, in one sentence?
   Sample answer

   A linear method that finds orthogonal directions of maximum variance and represents data by its projections onto the top few.

5. Why does PCA need standardized features?

   A good answer uses units.

   Sample answer

   Variance depends on units: income in euros varies in thousands, age in tens, so without scaling the income direction would dominate PC1 just because of its units.

6. What does PCA cost?
   Sample answer

   $O(nd^2)$ for the covariance matrix plus $O(d^3)$ for the eigendecomposition; projecting a point is $O(dk)$.

7. Give two pros and two cons.
   Sample answer

   Pros: removes redundancy with uncorrelated components, and fast with a clear "information kept" measure. Cons: linear only, and components are hard to interpret and scale-sensitive.

8. If you had to teach this to a friend in 2 minutes, what would you leave out, and why?
   Sample answer

   I'd leave out eigendecomposition details and the cost. I'd keep the camera-angle analogy, the 4-2-1-0.5 explained-variance example, and "keep the directions with the most spread".

External resources

• scikit-learn: PCA: explained variance and examples.
• Setosa: PCA explained visually: an interactive visual explanation.
• Wikipedia: Principal component analysis: derivations and applications.

Videos

• YouTube search: StatQuest PCA step by step: StatQuest's well-known PCA video.