Web Scraping & Automation

httpx

Requests' faster, more modern sibling, built for a Python that does several things at once.

Install it: pip install httpx

What does it do?

httpx does largely the same job as the older Requests library — fetching web pages and talking to APIs — but was built later to handle two things Requests wasn’t designed for: asynchronous code that can juggle many requests at once, and the newer HTTP/2 protocol many modern websites now use. Its everyday interface is deliberately almost identical to Requests, so switching between them feels familiar rather than like learning something new. Where Requests waits for one request to finish before starting the next by default, httpx can fire off hundreds simultaneously and handle whichever finishes first. It’s essentially Requests redesigned with a decade of hindsight about how the web actually works today.

See it in action

This code fetches data from a website over the internet, the same basic job as the Requests library, and prints the result.

import httpx

response = httpx.get("https://api.github.com")
print(response.status_code)
print(response.json())

Why would a non-developer care?

As websites and apps increasingly rely on fetching lots of small pieces of data at once rather than one big page, being able to make many requests concurrently instead of one after another can be the difference between a task taking seconds versus minutes. httpx brings that speed to Python without requiring anyone to relearn everything they already know from Requests. It matters most behind the scenes in apps and services that need to talk to many other services quickly.

Real-world examples

httpx was built by Encode, the same team behind the popular Django REST Framework and the Starlette web framework, and it’s become a common choice inside modern async Python web services like those built with FastAPI. Companies building APIs that need to call several other APIs at once, a common pattern in modern software, often reach for httpx specifically for its async support. Without it, developers needing both async support and HTTP/2 in Python would have to stitch together lower-level, less friendly tools.

Who uses it

Developers building modern async applications and APIs, especially those already using frameworks like FastAPI or Starlette.

How it compares to alternatives

httpx is essentially Requests plus async support and HTTP/2, aimed at replacing it in newer projects, though Requests still dominates in simpler synchronous scripts thanks to its enormous existing install base. aiohttp is httpx’s older async-focused rival, but httpx’s Requests-like interface is generally considered easier to pick up. For raw low-level connection handling, both ultimately rely on ideas pioneered by urllib3.

New to Python and want to actually try libraries like this yourself?

Find a beginner-friendly course

Related libraries