Web Scraping & Automation

urllib3

The unglamorous engine room that Requests, and much of the Python web, quietly runs on.

Install it: pip install urllib3

What does it do?

urllib3 handles the low-level plumbing of making HTTP connections from Python: opening connections, reusing them efficiently instead of starting fresh every time, handling retries, and managing secure encrypted connections. It’s more verbose and technical to use directly than Requests, because it’s meant to be a solid foundation that other, friendlier tools build on top of rather than something most people touch by hand. Its connection pooling, reusing the same open connection for multiple requests instead of constantly reconnecting, is a major reason software built on it is fast. Most people who benefit from urllib3 every day have never heard of it, because it operates one layer beneath what they actually interact with.

See it in action

This code opens a low-level internet connection, requests a webpage’s data, and prints back the status and raw content it received.

import urllib3

http = urllib3.PoolManager()
response = http.request("GET", "https://api.github.com")

print(response.status)
print(response.data)

Why would a non-developer care?

Every time a piece of Python software talks to the internet quickly and reliably without accidentally overwhelming a server with redundant new connections, there’s a good chance urllib3-style connection management is the reason. That efficiency isn’t visible to an end user, but it’s part of why some tools feel snappy while others built on cruder foundations feel sluggish. It’s infrastructure in the truest sense, invisible when it works, and the wider ecosystem quietly depends on it holding up.

Real-world examples

urllib3 is the actual engine that powers Requests underneath its friendlier interface, meaning nearly every Python program that uses Requests is also, indirectly, using urllib3. It’s also used directly inside pip, Python’s own package installer, to download packages from the internet. Without it, a huge share of Python’s networking code would need an entirely different low-level foundation to build on.

Who uses it

Library authors building higher-level tools who need reliable, low-level HTTP handling, rather than everyday developers working directly with it.

How it compares to alternatives

Requests is essentially a friendlier face built on top of urllib3, trading some low-level control for ease of use. httpx solves a similar low-level problem but adds native async support that urllib3 has only more recently started adding. Most developers never choose between them directly, since urllib3 usually arrives bundled inside something else.

Fun fact

urllib3 is used inside pip itself, meaning it’s involved in installing nearly every other Python package that gets downloaded, including, ironically, itself.

New to Python and want to actually try libraries like this yourself?

Find a beginner-friendly course

Related libraries