Velamuvi AIAcademy

Lesson 8 of 12 · Functions and Graphs

Non-Linear Functions

Meet the curved functions used in AI: quadratics, exponential and logarithmic functions, and the sigmoid, tanh and ReLU activation functions, with interactive graphs and Python.

  • 6 min read
  • Beginner
  • Reviewed
In this lesson you will learn
  • What makes a function non-linear
  • Quadratic functions and their parabola shape
  • Exponential and logarithmic functions as graphs
  • The activation functions sigmoid, tanh and ReLU
  • Why neural networks need non-linear functions

When a straight line is not enough

Real life is rarely a straight line. A child grows quickly, then slows down. A rumour spreads slowly, then explodes. A model's error drops fast at first, then levels off.

Quadratic functions

-4-2024-4-20246810
y = x²y = -0.5x² + 4
A positive a opens upwards, with a minimum. A negative a opens downwards, with a maximum.

Quadratics matter in AI because the most common way to measure a model's error, the squared error, is a parabola. Its single lowest point is the best parameter value, and training heads straight for it.

Try it yourselfPython
target = 3

for guess in [0, 1, 2, 3, 4, 5, 6]:
    squared_error = (guess - target) ** 2
    print(f"guess {guess}: squared error {squared_error:2} " + "#" * squared_error)

The little bar chart is a parabola lying on its side, with its minimum exactly at the right answer.

Exponential and logarithmic functions

You met these in the Exponents lesson. As graphs, they are mirror images of each other:

-4-20246-4-20246
y = eˣy = ln(x)y = x (the mirror)
eˣ shoots upwards. ln(x) grows ever more slowly. Each is the other reflected in the line y = x.

Activation functions

Here is where curves become essential to AI.

-4-2024-10123
sigmoidtanhReLU
Three activation functions. Sigmoid stays between 0 and 1, tanh between -1 and 1, and ReLU is 0 for negatives.
Try it yourselfPython
import math


def sigmoid(x):
    return 1 / (1 + math.exp(-x))


def relu(x):
    return max(0, x)


print(" x    sigmoid   tanh   relu")
for x in [-4, -1, 0, 1, 4]:
    print(f"{x:2}    {sigmoid(x):.3f}    {math.tanh(x):6.3f}   {relu(x)}")

Look at sigmoid: a large negative number gives almost 0, zero gives exactly 0.5, and a large positive number gives almost 1. A spam filter might output sigmoid(score) = 0.97, meaning "97% likely to be spam".

Why neural networks need curves

If every layer of a network were a straight-line function, stacking them would still only give a straight line, no matter how many layers you added. A straight line cannot separate cats from dogs, or understand a sentence.

Putting an activation function after each layer bends the lines. Many small bends, stacked together, can approximate almost any shape. That is the secret of deep learning's power.

Common mistakes

Practice exercise

Write the two most famous activation functions yourself.

Your solutionPython
import math

# 1. Write sigmoid(x) = 1 / (1 + e to the power -x), using math.exp.
# 2. Write relu(x), which returns x if it is positive and 0 otherwise.
# 3. For each score below, print the score, its sigmoid rounded to 2 places, and its relu.
scores = [-2, 0, 3]

# Output must be exactly:
# -2 0.12 0
# 0 0.5 0
# 3 0.95 3
Knowledge check

What does the sigmoid function do to any input number?

Summary

  • A non-linear function has a curved graph.
  • A quadratic makes a parabola. Squared error is a parabola, with its minimum at the right answer.
  • Exponential functions grow faster and faster. Logarithmic functions grow slower and slower.
  • Activation functions such as sigmoid, tanh and ReLU add curves to neural networks.
  • Without non-linear activations, a deep network would only ever draw a straight line.

You have finished the second module. Next: reading maths notation, starting with the sum symbol Σ.

Finished reading? Tick it off to track your progress.