Web Scraping With Python Requests: A Practical Guide

Requests is the library I use for the fetch step of almost every Python scraper. It sends an HTTP request and hands me back the status, headers, and body in a few lines. This guide covers the parts that matter for scraping: GET with query parameters, custom headers, sessions and cookies, POST, timeouts, and status handling. Every snippet was run with requests 2.34.2 on Python 3.13 in June 2026, so the output is real.

One note on the test target. I planned these calls against httpbin.org, but it was returning 503 from its CDN the whole time I wrote this. So I ran them against httpbingo.org, the official Go reimplementation of the same service. The endpoints match, with one quirk: httpbingo echoes header and query values as lists, which is why you see ['2'] instead of '2' below. The real fetch later uses quotes.toscrape.com, a site built for scraping practice.

What does the Requests library do in a scraper?

Requests handles the transport: it sends an HTTP request and returns the response. In a scraper it does one job, fetching the raw bytes of a page, and it does that job well. You install it with pip install requests, call requests.get(url), and read resp.text for the HTML or resp.json() for an API.

Requests stops at the raw HTML. It does not parse that HTML into fields and it does not run JavaScript. So the standard pattern is two libraries: Requests fetches, and a parser like BeautifulSoup extracts. I show that handoff at the end.

How do you send a GET request with query parameters?

Call requests.get(url, params=...) and pass a dict. Requests builds the query string and URL-encodes it for you, so you never hand-format ?page=2&sort=newest.

import requests

resp = requests.get(
    "https://httpbingo.org/get",
    params={"page": "2", "sort": "newest"},
)
print(resp.status_code)
print(resp.url)
print(resp.json()["args"])

The server echoed back the parameters it received, and resp.url shows the assembled query string:

200
https://httpbingo.org/get?page=2&sort=newest
{'page': ['2'], 'sort': ['newest']}

Building params as a dict keeps pagination clean: loop a page number, swap it into the dict, and let Requests handle the encoding. The status code lives on resp.status_code, and resp.json() parses a JSON body straight into a Python dict.

How do you set custom headers and a user agent?

Pass a headers dict to the request. The header that matters most for scraping is User-Agent, because the default Requests one announces a bot and many sites treat it differently.

import requests

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
    "Accept-Language": "en-US,en;q=0.9",
}
resp = requests.get("https://httpbingo.org/headers", headers=headers)
sent = resp.json()["headers"]
print(sent["User-Agent"])
print(sent["Accept-Language"])

The endpoint confirmed both headers arrived as sent:

['Mozilla/5.0 (Windows NT 10.0; Win64; x64)']
['en-US,en;q=0.9']

To see what Requests sends with no help from you, hit the user-agent endpoint with a bare call:

import requests

print(requests.get("https://httpbingo.org/user-agent").json())

{'user-agent': 'python-requests/2.34.2'}

That python-requests/2.34.2 string is the giveaway. Setting a realistic browser user agent is usually the first fix when a request that should work comes back blocked.

The Requests options that matter for scraping

These are the keyword arguments I pass most when scraping, with what each one controls. They work on requests.get, requests.post, and the Session methods.

Stacking a few reproduces a realistic browser request: requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, params={"page": 1}, timeout=10).

How do you keep cookies across requests with a Session?

Use requests.Session(). A session stores any cookies the server sets and sends them on later calls, and it reuses the connection. This is what you need for a login or anything that tracks state.

import requests

session = requests.Session()
session.headers.update({"User-Agent": "Mozilla/5.0"})

# server sets a cookie, the session stores it, the next call sends it back
session.get("https://httpbingo.org/cookies/set?token=abc123")
resp = session.get("https://httpbingo.org/cookies")
print("stored:", session.cookies.get_dict())
print("sent back:", resp.json())

The first call received a cookie, and the second call sent it back without any manual handling:

stored: {'token': 'abc123'}
sent back: {'cookies': {'token': 'abc123'}}

Setting session.headers once applies that user agent to every request from the session, so you set it in one place instead of on each call. For a multi-page crawl behind a login, a session is the difference between one request that works and a scraper that keeps its place.

How do you send a POST request?

Call requests.post(url, data=...) for a form-encoded body, the kind login and search forms submit. Pass json=... instead when an API expects JSON.

import requests

resp = requests.post(
    "https://httpbingo.org/post",
    data={"q": "web scraping", "page": "1"},
)
print(resp.status_code)
print(resp.json()["form"])

The endpoint echoed the form fields it parsed from the body:

200
{'page': ['1'], 'q': ['web scraping']}

The split is simple: data= sends application/x-www-form-urlencoded, and json= sends application/json and sets that content-type header for you. Match whichever the target form or API expects.

How do you set timeouts and handle status codes?

Pass timeout on every request and check the status before you parse. Without a timeout, Requests waits forever, so one unresponsive host can stall the whole run. This call sets a 2-second cap against an endpoint that delays 5 seconds:

import requests

try:
    # server waits 5s, our timeout is 2s
    requests.get("https://httpbingo.org/delay/5", timeout=2)
except requests.exceptions.Timeout:
    print("request timed out after 2s, as expected")

request timed out after 2s, as expected

For status, every response carries status_code and a boolean ok, and raise_for_status() turns a 4xx or 5xx into an exception you can catch:

import requests

resp = requests.get("https://httpbingo.org/status/404", timeout=10)
print("status_code:", resp.status_code)
print("ok:", resp.ok)
try:
    resp.raise_for_status()
except requests.exceptions.HTTPError as e:
    print("raised:", e)

status_code: 404
ok: False
raised: 404 Client Error: Not Found for url: https://httpbingo.org/status/404

Checking the status first saves you from feeding a block page or an error page to your parser and getting confusing empty results.

How do you fetch a real page and parse it?

Fetch with Requests, then hand the HTML to a parser. Here is a real fetch of a live practice site:

import requests

resp = requests.get("https://quotes.toscrape.com/page/1/", timeout=10)
print("status:", resp.status_code)
print("content-type:", resp.headers["Content-Type"])
print("bytes:", len(resp.content))

status: 200
content-type: text/html; charset=utf-8
bytes: 11064

That is the full job Requests does: a 200, the content type, and about 11 KB of HTML. To turn that HTML into data, pass resp.text to BeautifulSoup and select the elements you want:

import requests
from bs4 import BeautifulSoup

resp = requests.get("https://quotes.toscrape.com/page/1/", timeout=10)
soup = BeautifulSoup(resp.text, "html.parser")

for quote in soup.select(".quote")[:3]:
    text = quote.select_one(".text").get_text(strip=True)
    author = quote.select_one(".author").get_text(strip=True)
    print(f"{author}: {text}")

The parser pulled three quotes straight out of the page Requests fetched:

Albert Einstein: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
J.K. Rowling: “It is our choices, Harry, that show what we truly are, far more than our abilities.”
Albert Einstein: “There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”

That two-step, Requests for the fetch and a parser for the extraction, covers most static sites. From here you can widen the selectors or follow links. The CSS and XPath selectors guide goes deeper on the extraction half, and Scrapy wraps both steps into a framework if you outgrow a single script: see the Scrapy guide.

When Requests is not enough

Requests covers static HTML cleanly, and two limits push you past it. First, it does not run JavaScript, so a page that builds its content in the browser returns near-empty HTML to Requests. Second, at scale the requests start getting blocked: rate limits, IP bans, and CAPTCHAs land once you send enough traffic from one address.

The JavaScript gap calls for a headless browser. The blocking problem is where a scraper API like ChocoData takes over the fetch, rotating IPs and handling the block-handling layer behind one endpoint so your code keeps calling a single URL. The full decision tree for when to scale up lives in the web scraping guide.