Multi Layer Perceptrons

Multi-Layer Perceptrons (MLPs)

An MLP is a network of individual perceptrons, organized into interconnected sequential layers.

Anatomy of an MLP

In a standard feedforward MLP, data flows strictly in one direction (from input to output) through the following structure:

1. Input Layer: The initial nodes that receive the raw data features. (Technically, these aren’t “computing” neurons, just data placeholders).
2. Hidden Layers: All intermediate layers whose activations we don’t directly see as the final answer. These layers act as internal feature extractors, learning increasingly complex representations of the data.
3. Output / Outer Layer: The final layer whose computed output we actually observe and use for our prediction or classification.

Defining “Depth” in Deep Networks

The term “Deep Learning” is thrown around constantly, but mathematically, what makes a network “deep”?

Using standard graph theory, the depth of a network is defined as the length of the longest path from a root node (the source or input) to a leaf node (the sink or output).

• A single perceptron (Input -> Output) has a depth of 1.
• A network with one hidden layer (Input -> Hidden -> Output) has a depth of 2.

In deep learning literature, a network is formally considered “Deep” if its depth is strictly greater than 2. Therefore, a deep neural network must have at least 3 layers (e.g., an Input layer, at least two Hidden layers, and an Output layer). Depth means the number of layers with trainable parameters.

MLPs Approximate Functions

If we view a neural network mathematically, it is simply a massive composite function. You feed an input vector $X$ into the network, and it spits out an output vector $Y$. The remarkable thing about MLPs is the variety and complexity of functions they can emulate:

1. Boolean Functions: Just as Minsky and Papert proved that a 2-layer network can solve XOR, an MLP can perfectly compose and solve any complex boolean logic gate array.
2. Classification Boundaries: By chaining together multiple affine transformations and non-linear activations, MLPs can draw highly complex, non-linear decision boundaries around data clusters in multi-dimensional space.
3. Real-Valued Functions: Most importantly, MLPs can approximate continuous mathematical functions, mapping inputs to precise numerical outputs.

This leads us to one of the most famous theorems in artificial intelligence.

The Universal Approximation Theorem

The Universal Approximation Theorem was first proven for sigmoid activation functions by George Cybenko, and later generalized to multilayer architectures and other activations by Kurt Hornik.

Key Papers:

• Cybenko, G. (1989). “Approximation by superpositions of a sigmoidal function”
• Hornik, K. (1991). “Approximation capabilities of multilayer feedforward networks”

The theorem makes a profound guarantee:

A feedforward network with a linear output layer and at least one hidden layer containing a finite number of neurons (using a non-linear activation function) can approximate any continuous function on a compact subset of $\mathbb{R}^n$ to an arbitrary degree of precision.

In plain English: No matter how complex the relationship is between your inputs and outputs, there exists an MLP that can model it. You do not even need a “deep” network. The theorem proves that a shallow network with a depth of 2 (just one hidden layer) is theoretically sufficient, provided that layer is wide enough.

What are the Limitations?

If a shallow network can theoretically learn any function, why do we bother building deep networks, and why doesn’t machine learning perfectly solve every problem immediately?

The theorem is an existence proof—it guarantees the optimal network exists, but it doesn’t guarantee that we can easily find it. The limitations lie in practical execution:

1. Exponential Width: To approximate highly complex functions with only a single hidden layer, the number of neurons required in that layer scales exponentially. Deep networks are used because depth allows the network to compose hierarchical features (like edges $\rightarrow$ shapes $\rightarrow$ faces), which requires vastly fewer neurons than a single infinitely wide hidden layer.
2. The Optimization Problem: Just because the perfect weights exist doesn’t mean our training algorithm (like Gradient Descent) will find them. Training can easily fail due to vanishing gradients, getting stuck in local minima, or navigating poor loss landscapes.
3. Generalization vs. Memorization: The theorem guarantees the network can perfectly map the training data you provide. However, it does not guarantee that the network will generalize well to unseen data. A network with immense capacity might simply memorize the noise in your dataset (overfitting).
4. Data Availability: To accurately carve out the exact continuous function in a high-dimensional space, you need enough sample data to anchor the curve. We often lack sufficient data to approximate the function perfectly.

How MLPs Model Functions: Classification vs. Regression

If we view a neural network mathematically, it is simply a massive composite function. You feed an input vector $X$ into the network, and it spits out an output vector $Y$. But how does it physically map out these solutions?

1. Classification (Creating Boundaries)

In classification, the network’s goal is to draw a boundary between different clusters of data.

• When using piecewise linear activations (like ReLU or step functions), the network constructs jagged, intersecting hyperplanes—acting almost like a geodesic dome wrapping around the data.
• When using continuous, smooth activations (like Tanh or Sigmoid), the network constructs fluid, wavy, smooth boundaries that elegantly curve around the data clusters.

2. Regression (Predicting Real Values)

If we want to predict a continuous numerical curve (like a stock price over time), the Universal Approximation Theorem acts like a massive set of Lego blocks. We can visualize this clearly by looking at how a network builds a simple “square pulse.”

To approximate a highly complex, wavy continuous function, the network simply chains together hundreds of these isolated blocks side-by-side, adjusting the height of each pulse. If we use enough neurons to make the individual pulses incredibly narrow, the jagged stairs perfectly blur into a smooth, arbitrary curve.

The Network as a Function

When we think of network as a function, what it means is that the network is actually a universal map from the entire domain of individual values to the entire range of the output activation function. So if output activation is sigmoid, it means output can only lie between 0 and 1. This means if we decide the output activation is sigmoid, then we can compose an MLP that can model any function from a input space to the domain which lies between 0 and 1.

The MLP is a Universal Approximator for the entire class of functions (maps) it represents. where it represents a class of functions from some domain to some range or some co-domain.

Network Capacity and Architecture Sufficiency

Even though an MLP can approximate a function, it must possess sufficient structural “capacity” to do so. If the architecture is too small, it fundamentally cannot represent a highly complex pattern exactly.

For example, a specific classification pattern might require a minimum of 16 neurons in a single hidden layer to be captured perfectly. If you restrict that layer to fewer than 16 neurons, the network will fail to represent the pattern. Even if you add a second layer, if the first layer is bottlenecked (e.g., still having fewer than 16 neurons), the second layer would require a significantly larger number of neurons (e.g., 41 or more) just to compensate for the lost information.

A Note on Scaling Capacity: > How exactly do we measure this “capacity” as networks grow? In the 2017 paper “A Capacity Scaling Law for Artificial Neural Networks” by Gerald Friedland and Mario Krell, the authors derived two critical dimensions to predict perceptron network behavior: the lossless memory (LM) dimension and the MacKay (MK) dimension. Their crucial finding was that the capacity of a neural network scales strictly linearly with the total number of weights (parameters) in the network, establishing a hard mathematical upper bound on how much information a network can perfectly memorize and map.

Information Flow: Width, Depth, and Activation Functions

The choice of activation function directly dictates how information survives as it flows through the network, especially when dealing with narrow hidden layers.

• Threshold Activations (Hard Gates): A rigid threshold activation acts like a closed door. It either passes the information forward or it completely gates (blocks) it. If a narrow layer uses threshold activations, any information that fails to trigger the threshold is permanently lost and cannot be recovered by deeper layers.
• Continuous/Graded Activations: Functions like Sigmoid, Tanh, or ReLU produce a graded output. Instead of a hard block, they allow “missed” or partial information to trickle through to subsequent layers.

Because graded activations do not strictly destroy information, you can get away with using narrower layers (fewer neurons per layer). However, because the information is squeezed and diluted, the network requires greater depth (more layers) to fully process and untangle that graded information.

Depth vs. Width: The Exponential Advantages of Deep Networks

If a shallow network can theoretically learn any function, why do we bother building deep networks?

The Universal Approximation Theorem is an existence proof—it guarantees the optimal network exists, but it glosses over efficiency. Recent mathematical research has proven that depth provides an exponential advantage in representational capacity. Specifically, for piecewise linear activation functions like ReLU, adding depth increases the number of linear regions (how many ways the network can slice and fold the input space) exponentially, whereas adding width only increases it polynomially.

Key Papers on Optimal Depth:

• Eldan, R., & Shamir, O. (2016). “The Power of Depth for Feedforward Neural Networks”
• Telgarsky, M. (2016). “Benefits of depth in neural networks”

The Sawtooth Example

To see this exponential advantage in action, let’s look at an explicit mathematical example adapted from Matus Telgarsky’s paper.

Assume we have a 1-dimensional input ($x \in \mathbb{R}$) and we want the network to model a highly oscillating function—specifically, a “sawtooth” wave with $1024$ distinct teeth (linear segments).

1. The Shallow Network (Without Optimal Depth): In a shallow network with only 1 hidden layer, each ReLU neuron acts like a single hinge or “fold” in the function. To create a function with 1024 distinct teeth, you must explicitly encode each fold.

• You need exactly $1024$ independent neurons in that single hidden layer.
• Total required neurons: $1024$

2. The Deep Network (With Optimal Depth): Now consider a deep network. A single hidden layer of just 2 ReLU neurons can be configured to compute a simple triangle wave (1 tooth). If you feed that triangle wave into another layer of 2 ReLU neurons, the function folds over itself, producing 2 teeth. Feed it into another layer, and it folds again, producing 4 teeth.

Because of function composition ($f(f(f(x)))$), stacking layers doubles the number of teeth at every step. To get $2^{10} = 1024$ teeth, we only need to stack $10$ layers, each containing just 2 neurons.

• We need 10 layers $\times$ 2 neurons per layer.
• Total required neurons: $20$

The Conclusion: To approximate the exact same function, a shallow network required 1024 neurons, while a deep network required only 20 neurons. If we wanted $1,048,576$ teeth ($2^{20}$), the shallow network would need over a million neurons, while the deep network would need only 40.

If optimal depth is not achieved, the number of required perceptrons (the width) scales exponentially with the complexity of the function. Achieving optimal depth allows the network size to decrease exponentially, unlocking immense computational efficiency.

Feature Independence and The Curse of Dimensionality

A common misconception is that the number of neurons needed in a network grows exponentially with the raw number of input features (e.g., a 100x100 pixel image having 10,000 features). Mathematically, approximating a general function in high dimensions requires an exponentially large grid. This is known as the Curse of Dimensionality.

However, the number of perceptrons actually required grows exponentially only with the number of statistically independent features—often referred to as the data’s “intrinsic dimensionality” or the Manifold Hypothesis.

Key Paper:

• Poggio, T., Liao, Q., et al. (2017). “Why and when can deep-but not shallow-networks avoid the curse of dimensionality”

The Sensor Example

Imagine a small room with 100 temperature sensors scattered around it. You feed all 100 sensor readings (100 raw features) into an MLP to predict the room’s overall climate.

Because the sensors are close together, their readings are highly correlated. If a window opens, 50 sensors drop in temperature simultaneously. These features are not statistically independent. The entire room’s temperature might truly be governed by only 2 independent factors: (1) Is the heater on? (2) Is the window open?

The data from our sensors actually lies on a simple 2-dimensional manifold within a 100-dimensional space. When you feed this 100-dimensional vector into an MLP, the early layers of the network essentially “untwist” the crumpled paper, and the later layers learn the function on the flat 2D sheet.

Because the network only has to learn a 2-dimensional space, the required size of the neural network scales exponentially with the intrinsic features ($2^2$), not the raw inputs ($2^{100}$). If all 100 sensors were emitting perfectly independent, uncorrelated random noise, the network would instantly succumb to the curse of dimensionality, and the required perceptrons would skyrocket. Deep networks excel at naturally factoring these correlated hierarchies, bypassing the curse of dimensionality for real-world structured data like images, text, and audio.