Lesson 10 of 12 · Calling APIs From Python
Handling Errors and Timeouts
Learn to handle everything that can go wrong with an API call in Python: timeouts, connection errors, error status codes, raise_for_status and bad JSON, with a reusable safe function.
- 8 min read
- Beginner
- Reviewed
- The different ways an API call can fail
- What timeouts and connection errors are
- How raise_for_status() turns error codes into exceptions
- How to write one safe function that handles every failure
Things will go wrong
Your code can be perfect and an API call can still fail. The Wi-Fi drops. The server is restarting. The service is overloaded. Your key expires. A program that talks to APIs must expect failure and deal with it calmly.
There are three kinds of failure:
- No response at all: the request never reached the server, or the answer never came back.
- An error response: the server answered, but with a 4xx or 5xx status code.
- A strange response: the status is fine, but the body is not what you expected, such as text that is not JSON.
No response: connection errors and timeouts
In requests, both are exceptions. Remember try and except from Python for Beginners: the program jumps to except instead of crashing.
This example asks httpbin to wait 5 seconds before answering, but only allows 2:
import requests
try:
response = requests.get("https://httpbin.org/delay/5", timeout=2)
print("Got an answer:", response.status_code)
except requests.exceptions.Timeout:
print("The server took too long to answer.")
except requests.exceptions.ConnectionError:
print("Could not reach the server in time.")
except requests.exceptions.RequestException as error:
print("Some other problem:", type(error).__name__)On your own computer, this prints the timeout message. Inside a browser, a slow request is reported as a connection error instead. Catching both, as here, makes your code work everywhere.
Error responses: raise_for_status()
A 404 or 500 is not an exception in requests. The call succeeds and hands you an error response. You can check response.ok yourself, or ask requests to raise an exception for you:
import requests
for url in ["https://jsonplaceholder.typicode.com/posts/1", "https://jsonplaceholder.typicode.com/posts/99999"]:
try:
response = requests.get(url, timeout=10)
response.raise_for_status()
print("OK:", response.json()["title"][:30])
except requests.exceptions.HTTPError:
print("HTTP error:", response.status_code)raise_for_status() is handy because all your failures can then be handled in except blocks, in one place.
Strange responses: bad JSON
Sometimes the status is 200 but the body is an HTML page or broken text. Parsing it then fails. With the json module, the error is a ValueError. With requests, response.json() raises requests.exceptions.JSONDecodeError, which is a kind of ValueError too.
import json
bodies = ['{"temperature": 28.5}', "<html>Service temporarily unavailable</html>"]
for body in bodies:
try:
data = json.loads(body)
print("Temperature:", data["temperature"])
except ValueError:
print("The response was not valid JSON:", body[:30])Practising failure without the internet
To see all the cases reliably, here is a pretend fetch() function that fails in different ways depending on the address. Python has built-in TimeoutError and ConnectionError exceptions, which the pretend server raises just as a real network would:
def fetch(url):
# Pretend network, so every kind of failure can be shown on demand.
if "slow" in url:
raise TimeoutError("no answer within 5 seconds")
if "offline" in url:
raise ConnectionError("could not connect")
if "missing" in url:
return 404, '{"error": "not found"}'
if "broken" in url:
return 500, "Internal Server Error"
return 200, '{"message": "hello"}'
for url in ["/ok", "/slow", "/offline", "/missing", "/broken"]:
try:
status, body = fetch(url)
if status >= 400:
print(url, "-> server answered with an error:", status)
else:
print(url, "-> success:", body)
except TimeoutError:
print(url, "-> timed out")
except ConnectionError:
print(url, "-> could not connect")One safe function for everything
In a real project, you write the error handling once, in a helper function, and use it everywhere:
import requests
def get_json(url, params=None):
"""Return the JSON data from url, or None if anything went wrong."""
try:
response = requests.get(url, params=params, timeout=10)
response.raise_for_status()
return response.json()
except requests.exceptions.Timeout:
print("The server took too long to answer.")
except requests.exceptions.HTTPError:
print(f"The server answered with status {response.status_code}.")
except requests.exceptions.JSONDecodeError:
print("The answer was not valid JSON.")
except requests.exceptions.RequestException:
print("Could not reach the server. Check your connection.")
return NoneNote the order: the specific exceptions (Timeout, HTTPError, JSONDecodeError) come before the general RequestException. They are all kinds of RequestException, so if the general one came first, it would catch everything.
Common mistakes
Write a safe wrapper that turns every kind of failure into a clear message instead of a crash.
def fetch(url):
# Pretend network
if url == "/slow":
raise TimeoutError("too slow")
if url == "/offline":
raise ConnectionError("no connection")
if url == "/secret":
return 401, "Unauthorized"
return 200, "Welcome"
# Write safe_fetch(url). It calls fetch(url) and returns a short message:
# success (status below 400) -> "OK: <body>"
# error status -> "Error <status>"
# TimeoutError -> "Timed out"
# ConnectionError -> "No connection"
for url in ["/home", "/secret", "/slow", "/offline"]:
print(url, safe_fetch(url))
# Output must be exactly:
# /home OK: Welcome
# /secret Error 401
# /slow Timed out
# /offline No connection
With requests, what happens when the server answers 404 and you do not call raise_for_status()?
Summary
- API calls fail in three ways: no response, an error response or a strange response.
- Always set a timeout, and catch connection errors and timeouts with
tryandexcept. raise_for_status()turns 4xx and 5xx responses into anHTTPErrorexception.response.json()raises aValueErrorwhen the body is not valid JSON.- Write the error handling once, in a helper function, with specific exceptions before general ones.
Next: being polite to busy servers with rate limits and retries.
Finished reading? Tick it off to track your progress.