Start Learning Contact Us
Data Science with Python

Neural Network Basics

Deep Learning · Lesson 26 of 30

A neural network is a model made of many simple units arranged in layers. Deep learning means using networks with many layers. They are behind image recognition, speech recognition and every large language model.

A single neuron

A neuron multiplies each input by a weight, adds the results and a bias, and passes the total through an activation function.

import numpy as np

inputs = np.array([0.5, 0.8, 0.2])
weights = np.array([0.4, 0.3, -0.5])
bias = 0.1

total = inputs @ weights + bias
output = max(0.0, total)          # ReLU activation

print("weighted sum:", round(total, 3))
print("output:", round(output, 3))

With no activation function, this is exactly linear regression. The weights and the bias are the parameters that training adjusts.

Activation functions

An activation function adds non-linearity. Without it, any stack of layers would collapse into one linear calculation and could only fit straight-line relationships.

import numpy as np

z = np.array([-2.0, -0.5, 0.0, 0.5, 2.0])

relu = np.maximum(0, z)
sigmoid = 1 / (1 + np.exp(-z))
tanh = np.tanh(z)

print("input:  ", z)
print("ReLU:   ", relu)
print("sigmoid:", sigmoid.round(3))
print("tanh:   ", tanh.round(3))
Function Output range Typical use
ReLU 0 and above Hidden layers. The usual default
Sigmoid 0 to 1 Output for a yes or no probability
Tanh -1 to 1 Hidden layers in some older designs
Softmax Values that sum to 1 Output for several classes

A layer

A layer is many neurons that receive the same inputs. All of them can be calculated at once with a matrix multiplication.

import numpy as np

rng = np.random.default_rng(0)
x = np.array([0.5, 0.8, 0.2])          # 3 inputs

W = rng.normal(0, 0.5, size=(3, 4))    # 3 inputs to 4 neurons
b = np.zeros(4)

hidden = np.maximum(0, x @ W + b)
print("layer output:", hidden.round(3))
print("shape:", hidden.shape)

A network

A network passes the output of one layer into the next. This is the forward pass.

import numpy as np

rng = np.random.default_rng(1)

def layer(n_in, n_out):
    return rng.normal(0, 0.5, (n_in, n_out)), np.zeros(n_out)

W1, b1 = layer(3, 5)
W2, b2 = layer(5, 4)
W3, b3 = layer(4, 2)

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

x = np.array([0.5, 0.8, 0.2])
h1 = np.maximum(0, x @ W1 + b1)
h2 = np.maximum(0, h1 @ W2 + b2)
output = softmax(h2 @ W3 + b3)

print("class probabilities:", output.round(3))
print("parameters:", sum(a.size for a in (W1, b1, W2, b2, W3, b3)))
  • The input layer is the features.
  • Hidden layers transform them, each building on the one before.
  • The output layer gives the prediction.

The weights here are random, so the output is meaningless. Training, in the next lesson, finds weights that make it correct.

Why depth helps

Each layer can combine the features found by the previous one. In an image network, early layers respond to edges, later layers to shapes, and the last layers to whole objects. The network learns these features from data. Nobody designs them by hand, which is the main difference from the feature engineering you did in the machine learning module.

Counting parameters

A layer with n_in inputs and n_out neurons has n_in * n_out weights and n_out biases. Modern language models have billions of parameters arranged in the same basic way.

When to use a neural network

Data Usually best
Tables of numbers and categories Gradient-boosted trees
Images, audio, text Neural networks
Very little data Simple models

Neural networks need more data and computing power than classic models, and are harder to explain.

Practice

Change the middle layer of the network from 4 neurons to 16. How many parameters does the network have now?