But what is a neural network? | Deep learning chapter 1

Recognizing a digit like '3' from low-resolution images is an effortless task for the human brain, despite variations in pixel values and visual differences. The brain's visual cortex seamlessly identifies these as representing the same concept while distinguishing other digits as unique ideas. However, programming a computer to perform this recognition—analyzing 28x28-pixel grids and outputting corresponding numbers—is an immensely challenging problem.

Neural networks, pivotal in modern machine learning, are explored here not as buzzwords but through their mathematical structure. This introduction aims to demystify how neural networks function and what it means for them to "learn." Using the classic example of recognizing handwritten digits, this foundational approach focuses on a simple yet powerful vanilla form of neural network without advanced modifications. By understanding its basic structure and capabilities, one gains essential insights into more complex variants while appreciating its ability to perform tasks like digit recognition.

Neural networks are inspired by the brain, with neurons being fundamental units. A neuron is simply a holder of a number between 0 and 1, representing activation levels. For example, in an image input scenario with dimensions 28x28 pixels (784 total), each pixel corresponds to one neuron holding its grayscale value—0 for black and up to 1 for white.

A neural network begins with an input layer of 784 neurons, representing pixel brightness in images, and ends with an output layer of 10 neurons corresponding to digits. Hidden layers between these two are crucial for processing but their exact function remains complex and experimental. Activations from one layer influence the next, mimicking biological neuron behavior. Once trained on digit recognition, feeding image data into the network triggers specific activation patterns across layers until a final decision is made by identifying the brightest neuron in the output.

Layered neural networks aim to mimic human recognition by breaking down complex patterns into simpler components. For example, recognizing a digit like '9' involves identifying subcomponents such as loops or lines, with neurons in intermediate layers ideally corresponding to these features. The hope is that higher-level neurons activate for general patterns (e.g., any loop at the top), enabling subsequent layers to combine these activations into complete recognitions (like digits). This layered abstraction extends beyond image recognition, aiding tasks like speech parsing where raw audio transforms through syllables and words into abstract thoughts.

Neural networks detect edges by assigning weights to connections between neurons in different layers. These weights, visualized as a grid with positive and negative values, determine how pixel patterns are captured. By focusing on specific regions of an image and adjusting surrounding pixels' influence using weighted sums, the network identifies features like edges or loops. The sigmoid function transforms these sums into activations ranging from 0 to 1 for further processing. Biases adjust activation thresholds, ensuring neurons respond only when certain conditions are met.

A single neuron in a neural network requires calculating the sum of weights and biases before activation. In one layer, each neuron connects to 784 pixel neurons from the first layer, with individual weights and biases for every connection. A hidden layer containing 16 neurons results in 12,544 total connections (weights) plus an additional 16 biases just between two layers. Across all layers combined, this particular network involves nearly 13,000 adjustable parameters—weights and biases—that determine its behavior.

Learning in the context of computers involves adjusting numerous weights and biases to solve specific problems. Imagining manually setting these parameters highlights both the complexity and potential insights into how networks function. Developing an understanding of what each weight or bias represents allows for better troubleshooting when outcomes deviate from expectations, offering a foundation for structural improvements. Additionally, analyzing unexpected successful results can challenge assumptions and reveal alternative solutions.

Neural networks can be represented more compactly by organizing activations into vectors and weights into matrices. The weighted sum of activations corresponds to a matrix-vector product, with biases added as another vector for simplicity. Applying the sigmoid function element-wise completes the transition from one layer's activation to the next. This notation not only simplifies expressions but also optimizes computational efficiency through streamlined code.

Neurons are functions that process inputs from previous layers and output values between 0 and 1. The entire network operates as an intricate function, transforming 784 input numbers into 10 outputs using approximately 13,000 parameters like weights and biases to detect patterns. This complexity involves matrix-vector operations and the sigmoid activation function, enabling it to tackle digit recognition effectively.

Subscribing helps YouTube's algorithm recommend content from this channel, even if notifications aren't always received. Gratitude is expressed towards Patreon supporters for their contributions. Updates on the probability series are promised after a brief delay due to other projects.

Early neural networks utilized the sigmoid function to compress weighted sums into a range between zero and one, inspired by biological neuron activity. However, modern networks rarely use sigmoids due to training difficulties. Instead, ReLU (Rectified Linear Unit) has become prevalent for its simplicity and effectiveness in deep learning tasks. ReLU activates neurons only when inputs exceed a threshold; otherwise, it outputs zero—streamlining computations while improving performance.