Activation functions

Prerequisites

• The artificial neuron.
• Exponentials $e^z$.
• Derivatives (slopes), for the gradient discussion.

The analogy

How it works

The five functions: $\sigma(z) = \frac{1}{1 + e^{-z}}$, $\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$, $\text{ReLU}(z) = \max(0, z)$, $\text{LeakyReLU}(z) = \max(0.01z, z)$, $\text{softmax}(z)_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$.

Start with three scores by hand

Apply each function to $z = -2, 0, 2$. Use $e^{-2} \approx 0.135$ and $e^{2} \approx 7.389$. Predict, then open "Check".

Finish the picture: slopes matter

Training multiplies slopes (derivatives) layer after layer.

• Sigmoid slope $\sigma(z)(1 - \sigma(z))$ at $z = 0$ and $z = 2$. Predict both.
  Check

  0.25 at $z = 0$ (its maximum) and $0.881 \cdot 0.119 \approx 0.105$ at $z = 2$. Ten layers at 0.25 multiply to about $10^{-6}$.

• ReLU slope for positive $z$, and Leaky ReLU at $z = -2$. Predict both.
  Check

  ReLU: 1, so gradients pass unchanged. Leaky ReLU at −2: slope 0.01 and output −0.02, so a neuron is never completely "dead".

Full worked calculation

The three scores $z = -2, 0, 2$ pushed through every activation, with the arithmetic shown, then the derivatives. The derivatives decide how well gradients survive through many layers.

1. 1Sigmoid

   $$\sigma(2) = \frac{1}{1 + e^{-2}} = \frac{1}{1 + 0.1353} = 0.881, \qquad \sigma(-2) = 1 - \sigma(2) = 0.119$$

   The sigmoid is symmetric around 0.5: $\sigma(-z) = 1 - \sigma(z)$. You only need to compute one side.

2. 2Tanh

   $$\tanh(2) = \frac{e^{2} - e^{-2}}{e^{2} + e^{-2}} = \frac{7.389 - 0.135}{7.389 + 0.135} = 0.964$$

   Tanh is a stretched sigmoid ($\tanh z = 2\sigma(2z) - 1$) with outputs centred on 0, which tends to make the next layer's learning easier.

3. 3ReLU and Leaky ReLU

   $$\text{ReLU}(-2) = 0, \quad \text{ReLU}(2) = 2, \qquad \text{Leaky}(-2) = 0.01 \cdot (-2) = -0.02$$

   ReLU costs one comparison. The leaky version keeps a small slope for negatives so the neuron never becomes completely silent.

4. 4Softmax over the three scores

   $$e^{z} = [0.135,\; 1,\; 7.389], \quad \sum = 8.524 \;\Rightarrow\; \left[\tfrac{0.135}{8.524}, \tfrac{1}{8.524}, \tfrac{7.389}{8.524}\right] = [0.016,\; 0.117,\; 0.867]$$

   Softmax is different: it uses all the scores at once and returns probabilities that sum to 1. A gap of 2 between scores becomes a ratio of $e^2 \approx 7.4$ between probabilities.

5. 5Derivatives at z = 2

   $$\sigma'(2) = 0.881 \cdot 0.119 = 0.105, \qquad \tanh'(2) = 1 - 0.964^2 = 0.071, \qquad \text{ReLU}'(2) = 1$$

   At $z = 2$ the sigmoid and tanh are already flattening out, so their slopes are small. ReLU keeps a slope of exactly 1 for any positive input.

6. 6Why it matters: 5 layers deep

   $$\text{best case sigmoid: } 0.25^5 \approx 0.001, \qquad \text{ReLU (active path): } 1^5 = 1$$

   Backpropagation multiplies one derivative per layer. Even at its steepest the sigmoid's slope is 0.25, so after 5 layers the gradient is at most 0.1% of its size: the vanishing gradient. This is the main reason hidden layers use ReLU.

The rule in plain words

Hidden layers: use ReLU (or Leaky ReLU / GELU) by default; tanh for some recurrent nets.

Output layer: sigmoid for one yes/no probability, softmax for one-of-$K$ classes, nothing (linear) for regression.

The code

Read this code after you have computed the values by hand. It will make more sense.

double relu(double z)       { return z > 0 ? z : 0; }
double leaky_relu(double z) { return z > 0 ? z : 0.01 * z; }
double sigmoid(double z)    { return 1 / (1 + exp(-z)); }

void softmax(double z[], int k, double out[]) {
    double mx = z[0], sum = 0;
    for (int i = 1; i < k; i++) if (z[i] > mx) mx = z[i];
    for (int i = 0; i < k; i++) { out[i] = exp(z[i] - mx); sum += out[i]; }   /* shift for safety */
    for (int i = 0; i < k; i++) out[i] /= sum;
}

flowchart TD
  A{"Which layer?"} -->|hidden| H["ReLU (default) or Leaky ReLU / GELU"]
  A -->|output| B{"What do you predict?"}
  B -->|"one probability (yes/no)"| S["sigmoid"]
  B -->|"one of K classes"| SM["softmax"]
  B -->|"a real number"| L["linear (no activation)"]
          

Putting numbers on it

The derivatives that backpropagation multiplies through the layers:

Common mistakes

• Using softmax for multi-label problems (several classes can be true at once), where independent sigmoids are right.
• Using a sigmoid output with an unbounded regression target.
• Computing softmax without subtracting the maximum, which overflows for large scores.

Try it yourself

Did you know?

Key points to remember

If you remember one thing from each line, remember the because.

• Sigmoid suits a single probability because it maps any score into (0, 1).
• Softmax suits one-of-$K$ classes because its outputs are positive and sum to 1.
• ReLU trains deep nets well because its slope is 1 for positive inputs, so gradients don't shrink.
• Sigmoid and tanh saturate because their slopes approach 0 for large $|z|$.
• Leaky ReLU avoids dead neurons because it keeps a small slope for negative inputs.

Self-check questions

Answer these in order, out loud. They follow the page, so your answers become the script of your video.

1. How are sigmoid, tanh, ReLU and softmax like different knobs?
   Sample answer

   Sigmoid and tanh are dimmers that never fully switch, ReLU is an off/through switch, and softmax is a counter that splits 100% among several options.

2. Compute sigmoid, tanh and ReLU of $-2, 0, 2$, and softmax of $[-2, 0, 2]$.
   Sample answer

   Sigmoid 0.12, 0.5, 0.88; tanh −0.96, 0, 0.96; ReLU 0, 0, 2; softmax 0.016, 0.117, 0.867.

3. Why does softmax subtract mx before exponentiating?

   A good answer mentions overflow and differences.

   Sample answer

   Large scores would make $e^z$ overflow. Softmax only depends on differences between scores, so subtracting the maximum gives the same result safely.

4. So what is an activation function, in one sentence?
   Sample answer

   The non-linear function applied to a neuron's weighted sum, which lets networks represent non-linear patterns.

5. Why did ReLU replace sigmoid in hidden layers?

   A good answer compares slopes.

   Sample answer

   Sigmoid's slope is at most 0.25, so gradients shrink layer after layer; ReLU's slope is 1 for positive inputs, so gradients pass through deep networks.

6. What are the ranges and slopes at 0 of sigmoid and tanh?
   Sample answer

   Sigmoid: (0, 1), slope 0.25 at 0. Tanh: (−1, 1), slope 1 at 0.

7. Which activation goes on the output for regression, binary and multi-class problems?
   Sample answer

   Linear (none), sigmoid, and softmax.

8. If you had to teach this to a friend in 2 minutes, what would you leave out, and why?
   Sample answer

   I'd leave out Leaky ReLU and the softmax overflow trick. I'd keep the knob analogy, the values at −2, 0, 2, and "ReLU inside, sigmoid/softmax at the output".

External resources

• Wikipedia: Activation function: a table of functions and their properties.
• PyTorch docs: Non-linear activations: every activation available in PyTorch.
• Stanford CS231n: Activation functions: practical advice on choosing them.

Videos

• YouTube search: activation functions explained: pick a short visual overview.