The artificial neuron

Prerequisites

• Dot product: $w \cdot x = \sum_j w_j x_j$.
• The exponential function $e^x$.
• Logistic regression: one sigmoid neuron.

The analogy

How it works

Formula: $z = w^\top x + b$, then output $a = g(z)$ for an activation function $g$ (sigmoid, tanh, ReLU…).

Start with one neuron by hand

Inputs $x = [2, 3]$, weights $w = [0.5, -0.3]$, bias $b = 0.1$, sigmoid activation. Predict, then open "Check".

Finish the picture: what weights mean

Change the numbers and watch the output.

• Set $w_2 = +0.3$ instead. Predict $z$ and the output.
  Check

  $z = 1 + 0.9 + 0.1 = 2.0$; $\sigma(2) \approx 0.88$. A positive weight makes that input push the output up.

• Set the bias to $-1$ (original weights). Predict the output.
  Check

  $z = 0.1 - 1 = -0.9$; $\sigma(-0.9) \approx 0.29$. The bias shifts how easily the neuron "fires".

Full worked calculation

The neuron from the page, $x = [2, 3]$, $w = [0.5, -0.3]$, $b = 0.1$, computed one operation at a time, then with three activations, and finally: what happens to the output if one weight changes a little?

1. 1Multiply each input by its weight

   $$w_1 x_1 = 0.5 \cdot 2 = 1.0, \qquad w_2 x_2 = -0.3 \cdot 3 = -0.9$$

   A weight is how much the neuron cares about an input, and in which direction. A positive weight says "more of this input means fire more"; a negative one says the opposite.

2. 2Add them and the bias

   $$z = 1.0 + (-0.9) + 0.1 = 0.2 \qquad (\text{vector form: } z = w^\top x + b)$$

   The two inputs almost cancel out, and the bias tips the balance slightly positive. $z$ can be any real number; it is not yet the neuron's output.

3. 3Sigmoid activation

   $$\sigma(0.2) = \frac{1}{1 + e^{-0.2}} = \frac{1}{1 + 0.8187} = \frac{1}{1.8187} \approx 0.5498$$

   The sigmoid turns $z$ into a number between 0 and 1. $z = 0$ would give exactly 0.5; our slightly positive $z$ gives slightly more than 0.5, a weak "yes".

4. 4Other activations, same z

   $$\tanh(0.2) \approx 0.197, \qquad \text{ReLU}(0.2) = \max(0, 0.2) = 0.2$$

   Tanh maps into (−1, 1) and is centred on 0. ReLU passes positive values unchanged and cuts negatives to 0. The weighted sum is identical; only the final squashing differs.

5. 5How sensitive is the output?

   $$\frac{d\sigma}{dz} = \sigma(z)\,(1 - \sigma(z)) = 0.5498 \cdot 0.4502 \approx 0.2475$$

   Near $z = 0$ the sigmoid is at its steepest: a small change in $z$ changes the output by about a quarter as much. This number is exactly what backpropagation will use.

6. 6Nudge one weight

   $$w_1: 0.5 \to 0.6 \;\Rightarrow\; z = 0.2 + 0.1 \cdot 2 = 0.4, \qquad \sigma(0.4) \approx 0.5987$$

   Changing $w_1$ by 0.1 changes $z$ by $0.1 \times x_1 = 0.2$, because the weight is multiplied by an input of 2. The output rises from 0.550 to 0.599. Learning is choosing these nudges so outputs move towards the right answers.

The rule in plain words

Multiply each input by its weight and add them, then add the bias: $z = w^\top x + b$.

Pass $z$ through a non-linear activation $g$ to get the output $a = g(z)$. Learning means adjusting $w$ and $b$.

The code

Read this code after you have computed the neuron by hand. It will make more sense.

double neuron(double w[], double x[], int d, double b) {   /* sigmoid neuron */
    double z = b;
    for (int j = 0; j < d; j++) z += w[j] * x[j];             /* weighted sum + bias */
    return 1 / (1 + exp(-z));                                 /* activation */
}

flowchart LR
  X1["x1 = 2"] -->|"× 0.5"| S["sum + bias 0.1"]
  X2["x2 = 3"] -->|"× -0.3"| S
  S -->|"z = 0.2"| G["sigmoid"]
  G -->|"a = 0.55"| OUT["output"]
          

Putting numbers on it

The neuron in one line, with our numbers:

Common mistakes

• Forgetting the bias in $z$.
• Applying the activation to each input instead of to the sum.
• Thinking a neuron "fires" only at 1; with sigmoid the output is a graded value between 0 and 1.

Try it yourself

Did you know?

Key points to remember

If you remember one thing from each line, remember the because.

• A neuron is a weighted sum plus a bias because that lets each input count with its own learned importance.
• The bias shifts the threshold because it is added before the activation.
• The activation must be non-linear because stacked linear functions stay linear.
• One neuron can't do XOR because its decision boundary is a single straight line.
• Learning adjusts $w$ and $b$ because they are the only free numbers in the neuron.

Self-check questions

Answer these in order, out loud. They follow the page, so your answers become the script of your video.

1. How is a neuron like a judge listening to advisers?
   Sample answer

   Each adviser's opinion is weighted by trust, the judge adds a personal bias, and the total becomes a final score between 0 and 1.

2. Compute the neuron output for $x = [2, 3]$, $w = [0.5, -0.3]$, $b = 0.1$ with sigmoid.
   Sample answer

   $z = 0.2$, output $\sigma(0.2) \approx 0.55$.

3. Why does the code start z at b?

   A good answer says what $z$ must contain.

   Sample answer

   $z$ is the weighted sum plus the bias; starting at $b$ and adding each $w_j x_j$ builds exactly that in one loop.

4. So what is an artificial neuron, in one sentence?
   Sample answer

   A unit that computes a weighted sum of its inputs plus a bias and passes it through a non-linear activation.

5. Why must the activation be non-linear?

   A good answer composes two linear layers.

   Sample answer

   Two linear layers $W_2(W_1 x)$ equal one linear layer $(W_2 W_1) x$, so without non-linearity depth adds no power.

6. What does each part of the neuron do?
   Sample answer

   Weights scale the inputs, the bias shifts the threshold, the sum combines them, and the activation adds non-linearity.

7. Why can't a single neuron compute XOR?
   Sample answer

   Its boundary $w^\top x + b = 0$ is a straight line, and XOR's positive and negative cases can't be split by one line.

8. If you had to teach this to a friend in 2 minutes, what would you leave out, and why?
   Sample answer

   I'd leave out XOR and the history. I'd keep the judge analogy, the $z = 0.2 \to 0.55$ example, and "weighted sum + bias, then activation".

External resources

• 3Blue1Brown: But what is a neural network?: the visual introduction to neurons and layers.
• Neural Networks and Deep Learning, chapter 1: Michael Nielsen's free book: perceptrons and sigmoid neurons.
• Wikipedia: Artificial neuron: history and variants.

Videos

• 3Blue1Brown: But what is a neural network?: chapter 1 of the deep learning series.