Velamuvi AIAcademy

Lesson 10 of 12 · Calling APIs From Python

Handling Errors and Timeouts

Learn to handle everything that can go wrong with an API call in Python: timeouts, connection errors, error status codes, raise_for_status and bad JSON, with a reusable safe function.

  • 8 min read
  • Beginner
  • Reviewed
In this lesson you will learn
  • The different ways an API call can fail
  • What timeouts and connection errors are
  • How raise_for_status() turns error codes into exceptions
  • How to write one safe function that handles every failure

Things will go wrong

Your code can be perfect and an API call can still fail. The Wi-Fi drops. The server is restarting. The service is overloaded. Your key expires. A program that talks to APIs must expect failure and deal with it calmly.

There are three kinds of failure:

  1. No response at all: the request never reached the server, or the answer never came back.
  2. An error response: the server answered, but with a 4xx or 5xx status code.
  3. A strange response: the status is fine, but the body is not what you expected, such as text that is not JSON.

No response: connection errors and timeouts

In requests, both are exceptions. Remember try and except from Python for Beginners: the program jumps to except instead of crashing.

This example asks httpbin to wait 5 seconds before answering, but only allows 2:

Try it yourselfPython
import requests

try:
    response = requests.get("https://httpbin.org/delay/5", timeout=2)
    print("Got an answer:", response.status_code)
except requests.exceptions.Timeout:
    print("The server took too long to answer.")
except requests.exceptions.ConnectionError:
    print("Could not reach the server in time.")
except requests.exceptions.RequestException as error:
    print("Some other problem:", type(error).__name__)

On your own computer, this prints the timeout message. Inside a browser, a slow request is reported as a connection error instead. Catching both, as here, makes your code work everywhere.

Error responses: raise_for_status()

A 404 or 500 is not an exception in requests. The call succeeds and hands you an error response. You can check response.ok yourself, or ask requests to raise an exception for you:

Try it yourselfPython
import requests

for url in ["https://jsonplaceholder.typicode.com/posts/1", "https://jsonplaceholder.typicode.com/posts/99999"]:
    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()
        print("OK:", response.json()["title"][:30])
    except requests.exceptions.HTTPError:
        print("HTTP error:", response.status_code)

raise_for_status() is handy because all your failures can then be handled in except blocks, in one place.

Strange responses: bad JSON

Sometimes the status is 200 but the body is an HTML page or broken text. Parsing it then fails. With the json module, the error is a ValueError. With requests, response.json() raises requests.exceptions.JSONDecodeError, which is a kind of ValueError too.

Try it yourselfPython
import json

bodies = ['{"temperature": 28.5}', "<html>Service temporarily unavailable</html>"]

for body in bodies:
    try:
        data = json.loads(body)
        print("Temperature:", data["temperature"])
    except ValueError:
        print("The response was not valid JSON:", body[:30])

Practising failure without the internet

To see all the cases reliably, here is a pretend fetch() function that fails in different ways depending on the address. Python has built-in TimeoutError and ConnectionError exceptions, which the pretend server raises just as a real network would:

Try it yourselfPython
def fetch(url):
    # Pretend network, so every kind of failure can be shown on demand.
    if "slow" in url:
        raise TimeoutError("no answer within 5 seconds")
    if "offline" in url:
        raise ConnectionError("could not connect")
    if "missing" in url:
        return 404, '{"error": "not found"}'
    if "broken" in url:
        return 500, "Internal Server Error"
    return 200, '{"message": "hello"}'


for url in ["/ok", "/slow", "/offline", "/missing", "/broken"]:
    try:
        status, body = fetch(url)
        if status >= 400:
            print(url, "-> server answered with an error:", status)
        else:
            print(url, "-> success:", body)
    except TimeoutError:
        print(url, "-> timed out")
    except ConnectionError:
        print(url, "-> could not connect")

One safe function for everything

In a real project, you write the error handling once, in a helper function, and use it everywhere:

import requests


def get_json(url, params=None):
    """Return the JSON data from url, or None if anything went wrong."""
    try:
        response = requests.get(url, params=params, timeout=10)
        response.raise_for_status()
        return response.json()
    except requests.exceptions.Timeout:
        print("The server took too long to answer.")
    except requests.exceptions.HTTPError:
        print(f"The server answered with status {response.status_code}.")
    except requests.exceptions.JSONDecodeError:
        print("The answer was not valid JSON.")
    except requests.exceptions.RequestException:
        print("Could not reach the server. Check your connection.")
    return None

Note the order: the specific exceptions (Timeout, HTTPError, JSONDecodeError) come before the general RequestException. They are all kinds of RequestException, so if the general one came first, it would catch everything.

Common mistakes

Practice exercise

Write a safe wrapper that turns every kind of failure into a clear message instead of a crash.

Your solutionPython
def fetch(url):
    # Pretend network
    if url == "/slow":
        raise TimeoutError("too slow")
    if url == "/offline":
        raise ConnectionError("no connection")
    if url == "/secret":
        return 401, "Unauthorized"
    return 200, "Welcome"


# Write safe_fetch(url). It calls fetch(url) and returns a short message:
#   success (status below 400)  -> "OK: <body>"
#   error status                -> "Error <status>"
#   TimeoutError                -> "Timed out"
#   ConnectionError             -> "No connection"


for url in ["/home", "/secret", "/slow", "/offline"]:
    print(url, safe_fetch(url))

# Output must be exactly:
# /home OK: Welcome
# /secret Error 401
# /slow Timed out
# /offline No connection
Knowledge check

With requests, what happens when the server answers 404 and you do not call raise_for_status()?

Summary

  • API calls fail in three ways: no response, an error response or a strange response.
  • Always set a timeout, and catch connection errors and timeouts with try and except.
  • raise_for_status() turns 4xx and 5xx responses into an HTTPError exception.
  • response.json() raises a ValueError when the body is not valid JSON.
  • Write the error handling once, in a helper function, with specific exceptions before general ones.

Next: being polite to busy servers with rate limits and retries.

Finished reading? Tick it off to track your progress.