Web Scraping & Automation

requests

The library that made talking to the internet from Python feel like plain English.

Install it: pip install requests

What does it do?

Requests handles the unglamorous but essential job of having one computer program ask another for information over the internet — fetching a webpage, submitting a form, or pulling data from an online service. Before it existed, doing this in Python took several lines of fiddly, easy-to-get-wrong code; Requests reduced it to something close to ‘get this URL’ and ‘here’s what came back.’ It manages the technical handshake of HTTP, the protocol underneath essentially every website and app, so the person writing the code doesn’t have to think about it. It’s less a library than plumbing that a huge share of the Python internet quietly runs through.

See it in action

This code asks a website for information over the internet and then prints whether the request worked and what data came back.

import requests

response = requests.get("https://api.github.com")
print(response.status_code)
print(response.json())

Why would a non-developer care?

Nearly every time a Python program checks the weather, posts to social media, downloads a file, or talks to a payment processor, something like Requests is doing the legwork behind the scenes. It’s not flashy, but it’s the reason building software that connects to the outside world doesn’t require becoming a networking expert first. If you’ve ever used an app that pulls in live data, a library in this family made that possible.

Real-world examples

Requests was written by Kenneth Reitz in 2011 because he found Python’s built-in tools for this so painful, and it’s since been downloaded billions of times, making it one of the most depended-upon packages in the entire Python ecosystem. It underpins countless companies’ internal tools, data pipelines, and automation scripts, even when those companies never mention Python publicly. Losing it overnight would quietly break an enormous share of the world’s data-fetching scripts.

Who uses it

Practically every Python developer at some point, from beginners scraping a webpage to engineers building large-scale data pipelines.

How it compares to alternatives

Requests is synchronous and simple; httpx, built later by the Encode team, offers a near-identical interface but adds support for async code and the newer HTTP/2 protocol. urllib3 sits one level lower and actually powers Requests under the hood, but it’s more verbose to use directly. For most everyday tasks, Requests remains the default choice specifically because it asks so little of the person using it.

Fun fact

The official Requests documentation still carries the tagline ‘HTTP for Humans,’ a quiet jab at how unnecessarily complicated Python’s built-in urllib module felt by comparison.

New to Python and want to actually try libraries like this yourself?

Find a beginner-friendly course

Related libraries