Modern Perceptron and Activations

A Better Model: The Modern Perceptron

By slightly shifting our notation, we can represent the perceptron in a much more flexible way. Instead of treating the threshold $T$ as a separate logical condition, we can incorporate it directly into the input summation as a bias term, where the bias is simply the negative threshold ($b = -T$).

This allows us to describe the perceptron as a clean, two-step structural pipeline:

An Affine Combination: The structure first computes an affine combination of the inputs and weights to produce an intermediate value $z$. $$z = \sum_{i} w_i x_i + b$$
An Activation Function ($\theta$): This intermediate value $z$ is then passed through an activation function to determine the final output $y$.

In a basic, classic perceptron, this activation function is a very simple step-wise function: the output is 1 if the input is non-negative, and 0 otherwise.

$$ y = \begin{cases} 1 & \text{if } z \ge 0 \\ 0 & \text{otherwise} \end{cases} $$

However, by separating the affine input from the decision method, we are no longer limited to rigid boolean threshold outputs. Once we have the $\theta$ notation, we can substitute the step function with other continuous or differentiable activation functions, such as Sigmoid, Tanh, and ReLU.

Linear vs. Affine Combination

While often used interchangeably in casual discussion, there is a strict mathematical distinction between a linear combination and an affine combination.

Linear Combination: A sum of scaled inputs ($w_1 x_1 + w_2 x_2$). Geometrically, this restricts the resulting boundary line or hyperplane to always pass exactly through the origin (0,0).
Affine Combination: A linear combination plus an offset or translation ($w_1 x_1 + w_2 x_2 + b$). The addition of the bias term ($b$) allows the boundary to shift away from the origin, giving the model the freedom to fit patterns anywhere in the mathematical space.

Common Activation Functions

The separation of the affine transformation from the activation step unlocked the true potential of neural networks. By replacing the non-differentiable step function with continuous curves, researchers could use calculus (specifically, the chain rule) to update weights. This led to the widespread adoption of backpropagation.

Sigmoid (Logistic) Function

Paper: Learning representations by back-propagating errors by David Rumelhart, Geoffrey Hinton, and Ronald Williams, 1986.

The sigmoid function was critical to the first wave of deep learning. It was heavily utilized and popularized in the 1986 seminal paper by David Rumelhart et al. Because the step function has a derivative of zero almost everywhere (making gradient descent impossible), the sigmoid provided a smooth, differentiable alternative that “squashed” outputs into a valid probability range between 0 and 1.

Mathematical Definition: $$\sigma(z) = \frac{1}{1 + e^{-z}}$$

Derivative and Derivation: A beautiful property of the sigmoid function is that its derivative can be expressed entirely in terms of its own output, which makes computing gradients extremely fast. $$ \begin{aligned} \sigma’(z) &= \frac{d}{dz} (1 + e^{-z})^{-1} \\ &= -(1 + e^{-z})^{-2} (-e^{-z}) \\ &= \frac{e^{-z}}{(1 + e^{-z})^2} \\ &= \left( \frac{1}{1 + e^{-z}} \right) \left( \frac{e^{-z}}{1 + e^{-z}} \right) \\ &= \sigma(z) \left( \frac{(1 + e^{-z}) - 1}{1 + e^{-z}} \right) \\ &= \sigma(z) (1 - \sigma(z)) \end{aligned} $$

Tanh (Hyperbolic Tangent)

Paper: Efficient BackProp by Yann LeCun, 1998.

As neural networks grew slightly larger in the 1990s, researchers noticed that the sigmoid function often slowed down learning because its outputs were strictly positive (not zero-centered). Yann LeCun’s 1998 paper strongly advocated for the Tanh function. By shifting the curve to range between -1 and 1, the data remains zero-centered, which helps the gradients point closer to the true optimal direction during optimization, leading to faster convergence.

Mathematical Definition: $$\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$$

Derivative and Derivation: Like the sigmoid, the derivative of the Tanh function cleanly reduces to a formula based on its own output. Using the quotient rule: $$ \begin{aligned} \frac{d}{dz} \tanh(z) &= \frac{d}{dz} \left( \frac{e^z - e^{-z}}{e^z + e^{-z}} \right) \\ &= \frac{(e^z + e^{-z})(e^z + e^{-z}) - (e^z - e^{-z})(e^z - e^{-z})}{(e^z + e^{-z})^2} \\ &= \frac{(e^z + e^{-z})^2}{(e^z + e^{-z})^2} - \frac{(e^z - e^{-z})^2}{(e^z + e^{-z})^2} \\ &= 1 - \tanh^2(z) \end{aligned} $$

ReLU (Rectified Linear Unit)

Paper: Rectified Linear Units Improve Restricted Boltzmann Machines by Vinod Nair and Geoffrey Hinton, 2010.

Both Sigmoid and Tanh suffer from the “vanishing gradient” problem: for very high or low values of $z$, the curve goes completely flat, killing the gradient signal needed to train deep networks. ReLU became the gold standard after its use in the breakthrough 2012 AlexNet model. Because its derivative is a constant 1 for all positive inputs, it allows gradients to flow strongly through dozens of layers, enabling modern Deep Learning.

Mathematical Definition: $$\text{ReLU}(z) = \max(0, z)$$

Derivative: The derivative of ReLU is incredibly simple, making it highly computationally efficient. Note that strictly speaking, the derivative is undefined exactly at $z=0$, but in software implementations (like PyTorch or TensorFlow), it is arbitrarily assigned a value of 0. $$ \text{ReLU}’(z) = \begin{cases} 1 & \text{if } z > 0 \\ 0 & \text{if } z < 0 \end{cases} $$

Leaky ReLU

Paper: Rectifier Nonlinearities Improve Neural Network Acoustic Models by Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng, 2013.

While ReLU is incredibly effective, it suffers from the “Dying ReLU” problem: if a large gradient updates a weight such that the neuron’s input is always negative, that neuron will forever output 0. Because the gradient at 0 is also 0, it can never recover. Leaky ReLU solves this by allowing a small, non-zero gradient when the unit is inactive, keeping the neurons “alive” and updating.

Mathematical Definition: $$\text{LeakyReLU}(z) = \begin{cases} z & \text{if } z > 0 \\ \alpha z & \text{if } z \le 0 \end{cases}$$ (Where $\alpha$ is a small constant, typically 0.01).

Derivative: The derivative is trivial to compute, ensuring the computational efficiency of standard ReLU is maintained while preventing dead neurons. $$ \text{LeakyReLU}’(z) = \begin{cases} 1 & \text{if } z > 0 \\ \alpha & \text{if } z < 0 \end{cases} $$

GELU (Gaussian Error Linear Unit)

Paper: Gaussian Error Linear Units (GELUs) by Dan Hendrycks and Kevin Gimpel, 2016.

As AI architecture shifted heavily towards Transformers (the architecture powering models like BERT, GPT, and modern LLMs), researchers discovered that weighting inputs by their probability within a standard Gaussian distribution yielded better results. Unlike ReLU, GELU is a smooth, non-monotonic curve. It dips slightly below zero for small negative numbers before flattening out, essentially acting as a softer, probabilistic version of dropout and activation combined.

Mathematical Definition: $$\text{GELU}(z) = z \cdot \Phi(z)$$ (Where $\Phi(z)$ is the cumulative distribution function for Gaussian distribution. This is frequently approximated in code as $0.5 z (1 + \tanh[\sqrt{2/\pi}(z + 0.044715 z^3)])$).

Derivative: The derivative relies on the Gaussian probability density function $\phi(z)$. The non-zero derivative for small negative values gives it a similar anti-dying property to Leaky ReLU, but strictly bounded and mathematically smooth. $$ \text{GELU}’(z) = \Phi(z) + z \phi(z) $$