Transfer learning

Prerequisites

• CNNs.
• Dense layers and parameter counts.
• Overfitting on small datasets.

The analogy

How it works

Two strategies: Feature extraction: freeze the pretrained layers and train only a new head. Fine-tuning: then also unfreeze some top layers and train them with a small learning rate.

Start with the parameter count by hand

ResNet-18: about 11.7 million parameters; its last layer maps 512 features to 1,000 ImageNet classes. New task: cat vs dog. Predict, then open "Check".

Finish the picture: fine-tuning

After training the new head, you unfreeze the last block of ResNet-18 (about 8.4 million weights).

• Trainable parameters now. Predict roughly how many.
  Check

  About 8.4M + 1,026 ≈ 8.4M: far more capacity to adapt, but also more risk of overfitting 2,000 images.

• Learning rate. Predict a sensible choice.
  Check

  Much smaller than for the head, e.g. 10–100 times smaller, so the pretrained features change gently.

Full worked calculation

What actually trains during transfer learning: the new head. To keep the numbers readable, the frozen network outputs 3 features (a real ResNet-18 gives 512) for one photo of a dog: $f = [0.2, 1.5, 0.3]$. The new head has weights $W = \begin{bmatrix} 0.5 & -0.2 & 0.1 \\ -0.3 & 0.4 & 0.2 \end{bmatrix}$ (row 1 = cat, row 2 = dog), biases 0, learning rate 0.5.

1. 1Frozen features

   $$f = \text{backbone}(\text{photo}) = [0.2,\; 1.5,\; 0.3]$$

   The pretrained layers are not updated. They work as a fixed feature extractor: feature 2 might be "furry ears", learned from ImageNet's 1.2 million images.

2. 2Logits of the new head

   $$z_{\text{cat}} = 0.5 \cdot 0.2 - 0.2 \cdot 1.5 + 0.1 \cdot 0.3 = -0.17, \qquad z_{\text{dog}} = -0.06 + 0.6 + 0.06 = 0.60$$

   The head is a single dense layer: one dot product per class.

3. 3Softmax and loss

   $$p = \frac{[e^{-0.17}, e^{0.6}]}{e^{-0.17} + e^{0.6}} = [0.317,\; 0.683], \qquad L = -\ln 0.683 = 0.381$$

   The randomly initialised head already leans towards dog, but only weakly.

4. 4Gradient of the head only

   $$\frac{\partial L}{\partial z} = p - y = [0.317,\; -0.317], \qquad \frac{\partial L}{\partial W_{\text{dog}}} = -0.317 \cdot f = [-0.063,\; -0.475,\; -0.095]$$

   Each class row's gradient is its error times the features. The largest feature (1.5) gets the largest change. No gradient flows into the frozen backbone.

5. 5Update and check

   $$W_{\text{dog}} = [-0.268,\; 0.637,\; 0.248], \quad z = [-0.547,\; 0.977] \;\Rightarrow\; p_{\text{dog}} = 0.821, \; L = 0.197$$

   One step raised the dog probability from 0.68 to 0.82 and halved the loss. Only 6 numbers (8 with biases) changed.

6. 6At real scale

   $$512 \cdot 2 + 2 = 1{,}026 \text{ trainable} \quad \text{vs} \quad 11.2 \text{ million frozen} \;\Rightarrow\; 0.009\%$$

   Training under 0.01% of the weights is why transfer learning works with 2,000 photos and a laptop. Later, you can "fine-tune": unfreeze the top layers and train them with a much smaller learning rate.

The rule in plain words

Load a model pretrained on a large dataset, remove its task-specific output layer, add a new head for your classes, and freeze the rest. Train the head.

If you have enough data, unfreeze the top layers and fine-tune everything with a small learning rate. Preprocess inputs exactly as in pretraining.

The code

Read this code after you have counted the parameters by hand. It will make more sense.

long dense_params(long n_in, long n_out) { return n_in * n_out + n_out; }   /* weights + biases */

double trainable_share(long trainable, long total) {        /* fraction actually trained */
    return (double)trainable / total;
}

flowchart TD
  A{"How much labelled data do you have?"} -->|"little (hundreds to thousands)"| F["freeze backbone, train only the new head"]
  A -->|"more, or very different images"| T["train head, then fine-tune top layers with a small learning rate"]
  A -->|"huge dataset, very different domain"| S["consider training (or fully fine-tuning) the whole network"]
          

How costly is it?

The new head's size and its share of the model:

Common mistakes

• Forgetting to freeze the backbone, so 2,000 images overwrite millions of good weights.
• Fine-tuning with the same large learning rate as for the new head.
• Using different input preprocessing from the one used in pretraining.

Try it yourself

Did you know?

Key points to remember

If you remember one thing from each line, remember the because.

• Pretrained features transfer because early layers learn generic patterns like edges and textures.
• Only the head is replaced because the final layer is specific to the original classes.
• Freezing prevents overfitting because a small dataset then trains only a few thousand weights.
• Fine-tuning uses a small learning rate because large updates would destroy useful pretrained weights.
• Preprocessing must match because the pretrained weights expect inputs on the same scale.

Self-check questions

Answer these in order, out loud. They follow the page, so your answers become the script of your video.

1. How is transfer learning like a photographer learning to paint?
   Sample answer

   They reuse what they already know about light and composition and only learn the new brushwork; the network reuses general features and learns only the new head.

2. How many parameters does a new 512 → 2 head have, and what share of ResNet-18 is that?
   Sample answer

   1,026, about 0.01% of the roughly 11 million parameters.

3. Why does dense_params add n_out to n_in * n_out?

   A good answer mentions biases.

   Sample answer

   Each output neuron has one weight per input plus one bias, so the biases add $n_{\text{out}}$ parameters.

4. So what is transfer learning, in one sentence?
   Sample answer

   Reusing a model pretrained on a large dataset as the starting point for a new task, retraining only its head or fine-tuning it.

5. Why does transfer learning work with only 2,000 images?

   A good answer separates generic and specific features.

   Sample answer

   The backbone already extracts general features from 1.2 million images; with it frozen, the small dataset only has to train a tiny head, which it can do without overfitting.

6. What is the difference between feature extraction and fine-tuning?
   Sample answer

   Feature extraction freezes the pretrained layers and trains only the new head; fine-tuning also updates some or all pretrained layers with a small learning rate.

7. Why must preprocessing match the pretrained model?
   Sample answer

   The pretrained weights expect inputs with the same size and normalization; otherwise the features they compute become meaningless.

8. If you had to teach this to a friend in 2 minutes, what would you leave out, and why?
   Sample answer

   I'd leave out exact layer counts and preprocessing details. I'd keep the photographer analogy, the 1,026-weight head on a frozen ResNet, and "reuse, replace the head, fine-tune gently if you have more data".

External resources

• PyTorch tutorial: Transfer learning for computer vision: feature extraction and fine-tuning in code.
• Stanford CS231n: Transfer learning: when to freeze and when to fine-tune.
• Wikipedia: Transfer learning: overview and history.

Videos

• YouTube search: transfer learning explained: pick a short practical overview.