Introduction

Neural networks originally began as computational models of the brain and, more broadly, as models of human cognition.

Historically, the earliest model of cognition was associationism. Over time, this evolved into the connectionist model of the brain. Current artificial neural networks operate entirely as connectionist machines.

Multi-layer perceptrons (MLPs) are foundational connectionist computational models. While typically used as classification engines, MLPs are also capable of modeling continuous-valued functions. Ultimately, the most interesting AI tasks are simply complex functions that can be approximated by these networks.

Human Cognition

Cognition refers to the mental processes of acquiring, storing, retrieving, and using knowledge. This encompasses thinking, memory, perception, and problem-solving, allowing us to reason, plan, make decisions, and comprehend the world around us.

The earliest attempts to understand human cognition date back to 600 BC. At around 400 BC, Plato introduced foundational theories of cognition that would later form the basis of the theory of associationism.

Plato suggested that cognition is fundamentally based on forming associations.

The Theory of Associationism

In simple terms, the theory posits:

Our ability to think, recognize patterns, and act on those recognitions to draw inferences is entirely based on learned associations.

For instance: Lightning is usually followed by thunder. Upon observing this sequence, you form a mental association between lightning and thunder. Thereafter, if you see lightning, you expect to hear thunder; conversely, if you hear thunder, you infer that lightning must have struck nearby.

Machine learning can be understood through this exact lens of associations. At its core, an algorithm learns to associate a specific input with a specific output. While these associations can become arbitrarily complex, they remain associations nonetheless. Mathematical functions themselves are just formal associations.

The idea of associations is insightful, yet somewhat incomplete.
Simply stating that we form associations doesn’t explain the underlying mechanics. Even though the brain undeniably relies on these associations, critical questions remain:

• Where are these associations stored?
• How are they physically encoded?

To understand how associations are stored and inferences are made, we must look at the biological architecture of the human brain. By the mid-1800s, advancements in microscopy allowed scientists to examine brain slices, leading to the discovery that the brain is a massive network of interconnected neurons.

However, understanding the physical structure of the brain was only the first step; scientists still needed a computational model to explain how it processed information.

The Connectionist Framework: The First ANN Proposal

Modeling the brain was highly complex because the same biological organ performed a vast array of seemingly unrelated computational tasks.

The first computational model proposing how the brain operates was introduced in 1873 by Alexander Bain in his book Mind and Body. His core thesis was revolutionary for its time: the information lies within the connections.

Bain’s Key Ideas:

• Neural Groupings: Information is stored in the structural connections and how individual units are networked together.
• Making Memories: Bain proposed a mechanism for how a brain could learn to form these connections:

  “when two impressions concur, or closely succeed one another, the nerve-currents find some bridge or place of continuity, better or worse, according to the abundance of nerve matter available for the transition.”

• Remarkably, Bain was predicting Hebbian learning decades before it was formally articulated.

What Are the Units?

When modeling the brain, the fundamental unit of computation is the neuron.
A biological neuron consists of:

• Soma: The main body of the cell.
• Dentrites: The branching structures that receive incoming signals from other neurons. If the cumulative incoming signal exceeds a certain threshold, the neuron activates (fires).
• Axon: When the neuron fires, the electrical signal travels down the axon, which branches out to connect with the dendrites of other neurons.

Computational Models of the Neuron

Once the neuron was identified as the base unit, researchers required a formal mathematical model to simulate its behavior.

The McCulloch and Pitts Model (1943)

Paper: A Logical Calculus of the Ideas Immanent in Nervous Activity, Bulletin of Mathematical Biophysics, 5:115-137, 1943.

The figure above illustrates the mathematical representation of a neuron as defined by Warren McCulloch and Walter Pitts.
In this model, a neuron receives inputs through excitatory synapses. If the total signal coming down these connections exceeds a specific threshold, the neuron will fire. However, if any signal arrives via an inhibitory synapse (represented by the lower connection with a ring attached), it acts as an absolute veto, preventing the neuron from firing regardless of the excitatory inputs.

Even this highly simplified model was proven capable of computing a wide variety of Boolean logic functions.

Criticisms of the McCulloch-Pitts Model:

• They claimed their networks could compute a small class of functions, and that if provided with external memory (like a tape), their nets would be equivalent to Turing machines. This claim of Turing completeness was highly debated.
• They did not mathematically prove their own claims.
• Most importantly, the model lacked a learning mechanism; there was no way for the network to automatically adjust its connections to learn specific operations.

Donald Hebb and Hebbian Learning (1949)

Book: The Organization of Behavior: A Neuropsychological Theory, 1949.

Hebbian Learning provided the missing mechanism for neural adaptation:

“When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A’s efficiency, as one of the cells firing B, is increased.”

Often summarized simply as: “Neurons that fire together, wire together.”

If neuron $x_i$ repeatedly triggers neuron $y$, the synaptic connection between them strengthens. In mathematical terms, the weight ($w_i$) of the $i^{th}$ neuron’s input to the output neuron $y$ updates as follows:

$$ w_i = w_i + \eta x_i y $$

Numerous later modifications to basic Hebbian Learning introduced concepts like weight normalization and forgetting. For example, Generalized Hebbian Learning (also known as Sanger’s rule) distributes the contribution of an input incrementally over multiple outputs:

$$ w_{ij} = w_{ij} + \eta y_j \left( x_i - \sum_{k=1}^{j} w_{ik} y_k \right) $$

This conceptual formula forms the bedrock of many modern machine learning algorithms.

Frank Rosenblatt: The Perceptron (1958)

Paper: The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain, 1958.

Rosenblatt’s Perceptron improved upon the Hebbian learning method by introducing a mathematically robust architecture:

• Multiple inputs are combined linearly.
• Threshold logic: The neuron fires if the weighted sum of inputs exceeds a specific threshold.

$$ Y = \begin{cases} 1 & \text{if } \sum_{i} w_i x_i - T > 0 \\ 0 & \text{else} \end{cases} $$

Rosenblatt also formalized a supervised learning algorithm. The core idea was: to teach a perceptron a specific function, provide it with continuous examples of input-output pairs. Because the output is Boolean (it either fires or it doesn’t), the algorithm scales the weights by the difference between the desired output and the actual output.

$$ w = w + \eta (d(x) - y(x)) x $$

In this Sequential Learning equation:

• $d(x)$ is the desired target output for input $x$.
• $y(x)$ is the actual output produced by the perceptron.

The perceptron updates its weights whenever it makes a mistake, and Rosenblatt mathematically proved that this algorithm would successfully converge for any linearly separable dataset.

Minsky and Papert: Perceptrons Expanded (1969)

Book: Perceptrons: An Introduction to Computational Geometry, 1969 (Expanded Edition 1972).

While the Perceptron was a massive leap forward, Marvin Minsky and Seymour Papert demonstrated a critical limitation in 1969: a single-layer perceptron could not model the XOR (Exclusive OR) Boolean gate.

Because of this limitation, individual perceptrons were considered weak computational elements. To solve non-linearly separable problems like XOR, interconnected networks of elements were strictly required.

Solving XOR with a Multi-Layer Network

To solve XOR, Minsky and Papert theorized the addition of a “hidden layer” between the inputs and the output:

• The top unit in the hidden layer performs $X \lor Y$ (OR).
• The bottom unit in the hidden layer performs $\neg X \lor \neg Y$ (NAND).
• The final output unit combines these signals, performing $(X \lor Y) \land (\neg X \lor \neg Y)$, which successfully models $X \oplus Y$ (XOR).