Autoencoder

Prerequisites

• Dense layers.
• MSE loss.
• PCA: a linear autoencoder learns the same subspace.

The analogy

How it works

This example: linear encoder $z = W_e x$ with $W_e = \begin{bmatrix} 0.5 & 0 & 0.5 & 0 \\ 0 & 0.5 & 0 & 0.5 \end{bmatrix}$; decoder $\hat x = W_d z$ with $W_d = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 0 \\ 0 & 1 \end{bmatrix}$; loss = MSE between $x$ and $\hat x$.

Start with one input by hand

Input $x = [1, 0, 1, 0]$. Predict, then open "Check".

Finish the picture: an unusual input

Feed $x = [1, 1, 0, 0]$ to the same autoencoder.

• Code and reconstruction. Predict both.
  Check

  $z = [0.5, 0.5]$; $\hat x = [0.5, 0.5, 0.5, 0.5]$.

• Error, and what it tells you. Predict the MSE and its use.
  Check

  $(0.25 \cdot 4)/4 = 0.25$. A high error flags an input unlike the training data: the idea behind autoencoder anomaly detection.

Full worked calculation

The page's autoencoder, encoder $W_e = \begin{bmatrix} 0.5 & 0 & 0.5 & 0 \\ 0 & 0.5 & 0 & 0.5 \end{bmatrix}$ and decoder $W_d$ that copies each code value back to two positions, used as an anomaly detector on four inputs with a threshold of 0.1.

1. 1Encode

   $$z_1 = 0.5 x_1 + 0.5 x_3, \qquad z_2 = 0.5 x_2 + 0.5 x_4 \;\Rightarrow\; [1, 0, 1, 0] \mapsto [1, 0]$$

   The encoder squeezes 4 numbers into 2. Each code value is the average of two positions, so the code can only describe inputs where positions 1 and 3 (and 2 and 4) agree.

2. 2Decode

   $$\hat x = [z_1,\; z_2,\; z_1,\; z_2] = [1, 0, 1, 0]$$

   The decoder writes each code value back into the two positions it came from. For an input that follows the pattern, nothing is lost.

3. 3Reconstruction error

   $$\text{MSE} = \frac{1}{4}\sum_i (x_i - \hat x_i)^2 = 0$$

   The training loss of an autoencoder is exactly this error. A low value means "this input looks like the data I was trained on".

4. 4An input that breaks the pattern

   $$[1, 0, 0, 0] \mapsto z = [0.5, 0] \mapsto \hat x = [0.5, 0, 0.5, 0], \qquad \text{MSE} = \frac{0.25 + 0 + 0.25 + 0}{4} = 0.125$$

   The bottleneck has no way to say "position 1 but not position 3", so it averages them. The error reveals that this input is unusual.

5. 5The most unusual input

   $$[1, 1, 0, 0] \mapsto [0.5, 0.5] \mapsto [0.5, 0.5, 0.5, 0.5], \qquad \text{MSE} = \frac{4 \cdot 0.25}{4} = 0.25$$

   Every position is off by 0.5. This input contradicts the pattern twice, so its error is twice as large.

6. 6Decide with a threshold

   $$\text{MSE} > 0.1 \;\Rightarrow\; \text{anomaly}$$

   In practice the threshold is chosen from the errors on normal validation data, for example their 99th percentile. Here the two pattern-following inputs pass and the two others are flagged.

The rule in plain words

Train an encoder $z = f(x)$ to a small code and a decoder $\hat x = g(z)$ back to the input, minimising the reconstruction error between $x$ and $\hat x$.

Afterwards, use $z$ as a compact representation, or the reconstruction error as an anomaly score.

The code

Read this code after you have encoded and decoded by hand. It will make more sense.

double reconstruct_mse(double x[], double We[][4], double Wd[][2], double xh[]) {   /* 4 → 2 → 4 */
    double z[2];
    for (int k = 0; k < 2; k++) {                       /* encoder */
        z[k] = 0;
        for (int j = 0; j < 4; j++) z[k] += We[k][j] * x[j];
    }
    double err = 0;
    for (int j = 0; j < 4; j++) {                       /* decoder + squared error */
        xh[j] = Wd[j][0] * z[0] + Wd[j][1] * z[1];
        err += (x[j] - xh[j]) * (x[j] - xh[j]) / 4;
    }
    return err;
}

flowchart LR
  X["input x (4)"] --> ENC["encoder"]
  ENC --> Z["code z (2)"]
  Z --> DEC["decoder"]
  DEC --> XH["reconstruction x̂ (4)"]
  XH --> L["loss = ||x - x̂||²"]
  X --> L
          

Putting numbers on it

Training minimises the reconstruction error over the data:

Common mistakes

• Using a code as large as the input without other constraints, which lets the network learn the identity.
• Expecting a plain autoencoder to generate realistic new samples from random codes.
• Judging anomalies with a threshold set on test data instead of on normal training data.

Try it yourself

Did you know?

Key points to remember

If you remember one thing from each line, remember the because.

• The bottleneck forces compression because the code has fewer numbers than the input.
• No labels are needed because the training target is the input itself.
• Reconstruction error flags anomalies because inputs unlike the training data don't fit the learned code.
• Linear autoencoders match PCA because both find the best low-dimensional linear subspace under squared error.
• Plain autoencoders don't generate well because their code space isn't organised for random sampling.

Self-check questions

Answer these in order, out loud. They follow the page, so your answers become the script of your video.

1. How is an autoencoder like summarising an email in two bullet points?
   Sample answer

   If a colleague can rewrite the email well from just the bullets, the bullets captured what matters; the code plays the bullets' role.

2. Encode and decode $[1, 0, 1, 0]$ with the given weights.
   Sample answer

   Code $[1, 0]$, reconstruction $[1, 0, 1, 0]$, error 0.

3. Why does the code divide each squared difference by 4?

   A good answer mentions MSE.

   Sample answer

   It averages the squared errors over the 4 positions, giving the mean squared error.

4. So what is an autoencoder, in one sentence?
   Sample answer

   A network trained to reproduce its input through a smaller code, with an encoder and a decoder.

5. Why does $[1, 1, 0, 0]$ reconstruct badly?

   A good answer says what the code can express.

   Sample answer

   The code only stores "average of positions 1 and 3" and "average of 2 and 4", so this input becomes [0.5, 0.5] and decodes to a uniform blur.

6. What loss is used, and what variants exist?
   Sample answer

   Reconstruction error, usually MSE; variants include denoising, sparse and variational autoencoders.

7. How can an autoencoder detect anomalies?
   Sample answer

   Train it on normal data; inputs with unusually high reconstruction error are flagged as anomalies.

8. If you had to teach this to a friend in 2 minutes, what would you leave out, and why?
   Sample answer

   I'd leave out the variants and the PCA link. I'd keep the email-summary analogy, the 4 → 2 → 4 example with a good and a bad input, and "no labels, bottleneck, reconstruction error".

External resources

• Deep Learning book, chapter 14: Autoencoders: Goodfellow, Bengio and Courville, free online.
• Wikipedia: Autoencoder: variants and applications.
• PyTorch tutorial: Build a model: the building blocks for an encoder and decoder.

Videos

• YouTube search: autoencoders explained: pick a short visual explanation.