Multi-Layer Perceptrons and Feed-Forward Neural Networks

The perceptron and its learning rule, linear separability and the XOR problem, multi-layer feed-forward networks, activation functions, the forward pass and output layers (linear, sigmoid, softmax) with matching losses, backpropagation via the chain rule, training with (stochastic) gradient descent, counting parameters, the universal approximation theorem, vanishing gradients, initialisation and regularisation.

The perceptron

A single artificial neuron computes a weighted sum and thresholds it:

\hat y = \operatorname{sign}(\mathbf w^\top\mathbf x + b).

Perceptron learning rule (labels $\pm1$ ): for each misclassified example, $\mathbf w \leftarrow \mathbf w + \eta\,y\,\mathbf x$ , $b \leftarrow b + \eta\,y$ .

Convergence theorem: if the data is linearly separable, the rule finds a separating hyperplane in a finite number of updates (at most $(R/\gamma)^2$ for data radius $R$ and margin $\gamma$ ).
If the data is not separable (e.g. XOR), it never converges.

Minsky and Papert's observation: one layer cannot represent XOR; two can.

Multi-layer feed-forward networks

A multi-layer perceptron (MLP) stacks layers of neurons; information flows forward from inputs through hidden layers to outputs, with no cycles:

\mathbf h^{(1)} = g\big(W^{(1)}\mathbf x + \mathbf b^{(1)}\big), \quad \mathbf h^{(\ell)} = g\big(W^{(\ell)}\mathbf h^{(\ell - 1)} + \mathbf b^{(\ell)}\big), \quad \hat{\mathbf y} = o\big(W^{(L)}\mathbf h^{(L-1)} + \mathbf b^{(L)}\big).

A 3–4–2 network: (3·4 + 4) + (4·2 + 2) = 26 parameters.

Parameter count for layer sizes $n_0, n_1, \dots, n_L$ : $\sum_{\ell=1}^L(n_{\ell-1}n_\ell + n_\ell)$ .

Activation functions

Activation	$g(z)$	$g'(z)$	Notes
Sigmoid	$\frac{1}{1 + e^{-z}}$	$g(1 - g) \le 0.25$	outputs in $(0,1)$ ; saturates → vanishing gradients
tanh	$\frac{e^z - e^{-z}}{e^z + e^{-z}}$	$1 - g^2 \le 1$	zero-centred; still saturates
ReLU	$\max(0, z)$	0 or 1	default for hidden layers; cheap; "dying" units
Leaky ReLU	$\max(\alpha z, z)$	$\alpha$ or 1	keeps a small gradient for $z < 0$
Step (perceptron)	$[z > 0]$	0 almost everywhere	not trainable by gradients

Without non-linear activations, any depth collapses to a single linear map.

Output layers and losses

Task	Output activation	Loss
Regression	linear (identity)	mean squared error
Binary classification	sigmoid	binary cross-entropy
Multi-class classification	softmax $\frac{e^{z_k}}{\sum_je^{z_j}}$	categorical cross-entropy $-\log\hat y_{\text{true}}$

With softmax + cross-entropy (or sigmoid + binary cross-entropy), the gradient at the output pre-activations is simply $\hat{\mathbf y} - \mathbf y$ .

Backpropagation

Training minimises the loss by gradient descent; backpropagation computes all gradients efficiently by applying the chain rule backwards through the network, reusing intermediate results.

For layer $\ell$ with pre-activation $\mathbf z^{(\ell)} = W^{(\ell)}\mathbf a^{(\ell-1)} + \mathbf b^{(\ell)}$ and activation $\mathbf a^{(\ell)} = g(\mathbf z^{(\ell)})$ :

\boldsymbol\delta^{(L)} = \nabla_{\mathbf a^{(L)}}\mathcal L \odot o'(\mathbf z^{(L)}), \qquad \boldsymbol\delta^{(\ell)} = \big(W^{(\ell+1)\top}\boldsymbol\delta^{(\ell+1)}\big)\odot g'(\mathbf z^{(\ell)}),

\frac{\partial\mathcal L}{\partial W^{(\ell)}} = \boldsymbol\delta^{(\ell)}\,\mathbf a^{(\ell-1)\top}, \qquad \frac{\partial\mathcal L}{\partial\mathbf b^{(\ell)}} = \boldsymbol\delta^{(\ell)}.

Cost: one forward pass plus one backward pass, both about the same as a single evaluation of the network.

Training

Gradient descent $\boldsymbol\theta \leftarrow \boldsymbol\theta - \eta\nabla\mathcal L$ ; stochastic/mini-batch versions use a random batch per step (an epoch is one pass over the data).
Improvements: momentum, RMSProp, Adam (adaptive per-parameter step sizes); learning-rate schedules.
Initialisation matters: all-zero weights make hidden units identical (symmetry is never broken); use small random weights scaled to the layer size (Xavier/Glorot for tanh, He for ReLU).
The loss is non-convex, but in practice SGD finds good solutions.

Problems and remedies

Problem	Remedy
Vanishing gradients (deep sigmoid/tanh nets)	ReLU, careful initialisation, batch normalisation, residual connections
Exploding gradients	gradient clipping, smaller learning rate
Overfitting	weight decay (L2), dropout (randomly zero units during training), early stopping on validation loss, data augmentation, more data
Slow convergence	input normalisation, Adam, batch normalisation

Expressive power

Universal approximation theorem: a feed-forward network with a single hidden layer of sufficiently many units and a non-polynomial activation can approximate any continuous function on a compact set to arbitrary accuracy. Depth makes many functions far more parameter-efficient to represent.