Regularization (L1 and L2)

Prerequisites

• Bias–variance trade-off.
• Linear regression and its squared loss.
• Absolute values and squares.

The analogy

How it works

Assumption for the numbers: features are standardized and uncorrelated (orthonormal). Then ridge and lasso have simple closed forms per weight; with correlated features the effect is similar but not exactly these numbers.

Start with four weights by hand

Unregularized weights $w = [4.0, -3.0, 0.5, 0.1]$. Ridge with λ = 1 divides each weight by $1 + \lambda$. Lasso with λ = 0.6 moves each weight towards 0 by 0.6, stopping at 0. Predict, then open "Check".

Finish the picture: tuning λ

Try lasso with other strengths on the same weights $[4.0, -3.0, 0.5, 0.1]$.

• λ = 0.2. Predict the weights.
  Check

  $[3.8, -2.8, 0.3, 0]$: only the tiniest weight is removed.

• λ = 5. Predict the weights and what that means for the model.
  Check

  All four become 0: the model always predicts the same value. Too much regularization means underfitting; λ is chosen by cross-validation.

Full worked calculation

The page's unregularized weights $\hat w = [4.0, -3.0, 0.5, 0.1]$, with orthonormal features (the case the code assumes). There, the data-fit term becomes $\tfrac12 \lVert w - \hat w \rVert^2$ plus a constant, so each weight can be solved separately. We derive both shrink rules, compute the cost before and after, and check what happens to a prediction.

1. 1Ridge for one weight

   $$\frac{d}{dw}\Big[\tfrac12 (w - 4)^2 + \tfrac{\lambda}{2} w^2\Big] = (w - 4) + \lambda w = 0 \;\Rightarrow\; w = \frac{4}{1 + \lambda} = 2$$

   Setting the derivative to zero balances the pull towards the data ($w - 4$) against the pull towards 0 ($\lambda w$). With $\lambda = 1$ they meet halfway.

2. 2Ridge for all four

   $$w = \frac{\hat w}{2} = [2.0,\; -1.5,\; 0.25,\; 0.05]$$

   Every weight is divided by the same factor. Big or small, nothing reaches exactly 0, so all four features stay in the model.

3. 3Ridge pays off

   $$\underbrace{\tfrac12(4 + 2.25 + 0.0625 + 0.0025)}_{\text{fit cost} = 3.16} + \underbrace{\tfrac12(4 + 2.25 + 0.0625 + 0.0025)}_{\text{penalty} = 3.16} = 6.32 < \tfrac12 \cdot 25.26 = 12.63$$

   Staying at $\hat w$ costs nothing in fit but 12.63 in penalty. Moving halfway costs 3.16 in fit and saves far more in penalty: the optimiser happily gives up a little accuracy on the training data.

4. 4Lasso for a big weight

   $$\frac{d}{dw}\Big[\tfrac12 (w - 4)^2 + 0.6\,|w|\Big] = (w - 4) + 0.6 = 0 \;\Rightarrow\; w = 3.4$$

   For $w > 0$ the derivative of $|w|$ is just 1, so the L1 pull is a constant 0.6, whatever the size of the weight. Big weights move by exactly $\lambda$. For $-3.0$ the same logic gives $-2.4$.

5. 5Lasso for a small weight

   $$w = 0.5: \quad \frac{d}{dw}\Big[\tfrac12 (w - 0.5)^2 + 0.6\,w\Big]_{w = 0^+} = -0.5 + 0.6 = 0.1 > 0$$

   Right at 0 the cost is already increasing towards the right: the data's pull (0.5) is weaker than the constant penalty (0.6). The minimum is exactly $w = 0$, so the feature is switched off. The same happens to 0.1.

6. 6Lasso pays off

   $$\underbrace{\tfrac12(0.36 + 0.36 + 0.25 + 0.01)}_{0.49} + \underbrace{0.6\,(3.4 + 2.4 + 0 + 0)}_{3.48} = 3.97 < 0.6 \cdot 7.6 = 4.56$$

   Zeroing the two small weights costs only 0.13 in fit and saves 0.36 in penalty, so the optimiser prefers a simpler model with 2 features instead of 4.

7. 7Effect on a prediction

   $$x = [1, 1, 1, 1]: \quad \hat w \cdot x = 1.6, \qquad w_{\text{ridge}} \cdot x = 0.8, \qquad w_{\text{lasso}} \cdot x = 1.0$$

   Regularized models make smaller, more cautious predictions. That costs a little accuracy on training data and usually gains more on new data, where large weights would amplify noise.

The rule in plain words

Add a penalty to the training loss: L2 adds $\lambda \sum w_j^2$ (shrinks all weights smoothly), L1 adds $\lambda \sum |w_j|$ (shrinks and zeroes weights).

Choose λ by cross-validation: too small barely helps, too large underfits. Standardize features first so the penalty treats them fairly.

The code

Read this code after you have shrunk the weights by hand. It will make more sense.

double ridge_shrink(double w, double lambda) {      /* L2, orthonormal features */
    return w / (1 + lambda);                         /* shrink, never exactly 0 */
}

double lasso_shrink(double w, double lambda) {      /* L1: soft-thresholding */
    if (w > lambda) return w - lambda;
    if (w < -lambda) return w + lambda;
    return 0;                                        /* small weights switched off */
}

flowchart TD
  A{"Is the model overfitting?"} -->|no| N["no regularization needed (or a little)"]
  A -->|yes| B{"Do you suspect many useless features?"}
  B -->|yes| L1["L1 / lasso: removes features"]
  B -->|"no, many small effects"| L2["L2 / ridge: shrinks all weights"]
  B -->|"both, correlated features"| EN["Elastic net: L1 + L2"]
          

Putting numbers on it

The two regularized losses (with the convention that gives the per-weight rules above):

With orthonormal features, minimising them gives $w_{L2} = w / (1 + \lambda)$ and $w_{L1} = \text{sign}(w) \max(|w| - \lambda, 0)$.

Common mistakes

• Penalising the bias (intercept) term, which shifts all predictions for no reason.
• Forgetting to standardize features before applying the penalty.
• Choosing λ on the test set instead of with cross-validation.

Try it yourself

Did you know?

Key points to remember

If you remember one thing from each line, remember the because.

• Regularization reduces overfitting because it makes large, noise-chasing weights expensive.
• L1 sets weights to exactly zero because it subtracts a constant amount, stopping at zero.
• L2 never zeroes weights because dividing a nonzero number never gives zero.
• λ must be tuned because too little doesn't help and too much underfits.
• Features must be standardized because the penalty depends on the scale of each weight.

Self-check questions

Answer these in order, out loud. They follow the page, so your answers become the script of your video.

1. How is regularization like a fee on a travel budget?
   Sample answer

   A fee per item makes you drop marginal items (L1); a fee that grows with size makes you buy smaller versions of everything (L2).

2. Apply ridge (λ = 1) and lasso (λ = 0.6) to $[4.0, -3.0, 0.5, 0.1]$.
   Sample answer

   Ridge: $[2.0, -1.5, 0.25, 0.05]$. Lasso: $[3.4, -2.4, 0, 0]$.

3. Why does lasso_shrink return 0 when $-\lambda \le w \le \lambda$?

   A good answer compares the gain and the cost.

   Sample answer

   For small weights the reduction in error is smaller than the L1 cost of keeping them, so the best value is exactly 0.

4. So what is regularization, in one sentence?
   Sample answer

   Adding a penalty on the size of the model's weights to its loss, to discourage overfitting.

5. Why does L1 produce sparse models but L2 doesn't?

   A good answer uses the per-weight update rules.

   Sample answer

   L1 shrinks each weight by a fixed amount and stops at 0, so small weights land exactly on 0. L2 shrinks by a factor, so weights get smaller but never reach 0.

6. What do the L1 and L2 penalties look like?
   Sample answer

   L1: $\lambda \sum |w_j|$. L2: $\lambda \sum w_j^2$ (often with a factor of ½).

7. How do you choose λ?
   Sample answer

   By cross-validation: try several values and keep the one with the best validation score.

8. If you had to teach this to a friend in 2 minutes, what would you leave out, and why?
   Sample answer

   I'd leave out the exact loss conventions and elastic net. I'd keep the budget analogy, the four-weight example, and "penalty on weights; L2 shrinks, L1 shrinks and zeroes; tune λ".

External resources

• scikit-learn: Ridge and Lasso: both models with examples.
• Wikipedia: Lasso: soft-thresholding and the geometry of L1.
• Google ML Crash Course: Regularization: L2 and λ explained visually.

Videos

• YouTube search: StatQuest ridge and lasso: StatQuest's two clear videos.