Lesson 8 of 12 · Functions and Graphs
Non-Linear Functions
Meet the curved functions used in AI: quadratics, exponential and logarithmic functions, and the sigmoid, tanh and ReLU activation functions, with interactive graphs and Python.
- 6 min read
- Beginner
- Reviewed
- What makes a function non-linear
- Quadratic functions and their parabola shape
- Exponential and logarithmic functions as graphs
- The activation functions sigmoid, tanh and ReLU
- Why neural networks need non-linear functions
When a straight line is not enough
Real life is rarely a straight line. A child grows quickly, then slows down. A rumour spreads slowly, then explodes. A model's error drops fast at first, then levels off.
Quadratic functions
Quadratics matter in AI because the most common way to measure a model's error, the squared error, is a parabola. Its single lowest point is the best parameter value, and training heads straight for it.
target = 3
for guess in [0, 1, 2, 3, 4, 5, 6]:
squared_error = (guess - target) ** 2
print(f"guess {guess}: squared error {squared_error:2} " + "#" * squared_error)The little bar chart is a parabola lying on its side, with its minimum exactly at the right answer.
Exponential and logarithmic functions
You met these in the Exponents lesson. As graphs, they are mirror images of each other:
Activation functions
Here is where curves become essential to AI.
import math
def sigmoid(x):
return 1 / (1 + math.exp(-x))
def relu(x):
return max(0, x)
print(" x sigmoid tanh relu")
for x in [-4, -1, 0, 1, 4]:
print(f"{x:2} {sigmoid(x):.3f} {math.tanh(x):6.3f} {relu(x)}")Look at sigmoid: a large negative number gives almost 0, zero gives exactly 0.5, and a large positive number gives almost 1. A spam filter might output sigmoid(score) = 0.97, meaning "97% likely to be spam".
Why neural networks need curves
If every layer of a network were a straight-line function, stacking them would still only give a straight line, no matter how many layers you added. A straight line cannot separate cats from dogs, or understand a sentence.
Putting an activation function after each layer bends the lines. Many small bends, stacked together, can approximate almost any shape. That is the secret of deep learning's power.
Common mistakes
Write the two most famous activation functions yourself.
import math # 1. Write sigmoid(x) = 1 / (1 + e to the power -x), using math.exp. # 2. Write relu(x), which returns x if it is positive and 0 otherwise. # 3. For each score below, print the score, its sigmoid rounded to 2 places, and its relu. scores = [-2, 0, 3] # Output must be exactly: # -2 0.12 0 # 0 0.5 0 # 3 0.95 3
What does the sigmoid function do to any input number?
Summary
- A non-linear function has a curved graph.
- A quadratic makes a parabola. Squared error is a parabola, with its minimum at the right answer.
- Exponential functions grow faster and faster. Logarithmic functions grow slower and slower.
- Activation functions such as sigmoid, tanh and ReLU add curves to neural networks.
- Without non-linear activations, a deep network would only ever draw a straight line.
You have finished the second module. Next: reading maths notation, starting with the sum symbol Σ.
Finished reading? Tick it off to track your progress.