Python HTTP requests for scraping

A scraper begins with a request, not a regular expression. The request says which URL you want, who is asking, and how long the client should wait. This lesson builds that request object and explains the failures you must handle before parsing anything. You will also encode a query, inspect HTTPError, and bound retries.

A request has more than a URL

urllib.request.Request stores the URL and optional headers. A descriptive User-Agent tells the server who is calling. Building the request does not send it; urlopen performs the network call in a normal Python environment.

from urllib.request import Request

request = Request(
    "https://example.com/docs",
    headers={"User-Agent": "PyPracticeBot/1.0"},
)
print(request.full_url)
print(request.get_header("User-agent"))

Build before you send

Plan for a slow or broken server

Networks fail and pages move. A timeout bounds the wait, while the status code tells you whether the response is usable. Treat 2xx as success, 3xx as a redirect to resolve, and 4xx or 5xx as cases that need a decision rather than blind retries.

status = 404
if 200 <= status < 300:
    result = "ok"
elif 300 <= status < 400:
    result = "redirect"
else:
    result = "error"
print(result)

Branch on the status

Keep fetching separate from parsing

Write parsing as a function that accepts HTML text. Then you can test it with a saved fixture and no network. This boundary also makes it obvious where rate limiting and caching belong.

from html.parser import HTMLParser

class Links(HTMLParser):
    def __init__(self):
        super().__init__()
        self.links = []

    def handle_starttag(self, tag, attrs):
        if tag == "a":
            href = dict(attrs).get("href")
            if href:
                self.links.append(href)

def parse_links(html):
    parser = Links()
    parser.feed(html)
    return parser.links

saved = '<a href="/docs">Docs</a><a href="/about">About</a>'
print(parse_links(saved))

A parser you can test offline

Encode the query before building the request

urlencode escapes spaces, ampersands, and non-ASCII text. Build the URL from encoded parts instead of concatenating raw user input. A descriptive User-Agent and an Accept header also make the request easier to identify and debug.

from urllib.parse import urlencode
from urllib.request import Request

query = urlencode({"q": "green tea", "page": 2})
url = "https://example.com/search?" + query
request = Request(url, headers={"User-Agent": "PyPracticeGift/1.0"})
print(request.full_url)
print(request.get_header("User-agent"))

A URL assembled from encoded pieces

Treat HTTPError as a response

urllib raises HTTPError for 4xx and 5xx statuses, but the exception still carries a status code, headers, and a readable body. Catch it when the error page contains useful information; do not catch every network exception and pretend the request succeeded.

from io import BytesIO
from urllib.error import HTTPError

error = HTTPError(
    "https://example.com/missing",
    404,
    "Not Found",
    {"Content-Type": "text/plain"},
    BytesIO(b"no such page"),
)
print(error.code, error.reason)
print(error.read().decode("utf-8"))

The failure body is still data

Bound retries and honor Retry-After

Retry only errors that are likely to be temporary, such as 429 and 503. Retry-After may be a number of seconds or an HTTP date; cap it so a server cannot make your worker sleep for hours. Always combine retries with a maximum attempt count.

def retry_delay(status, retry_after, cap=300):
    if status not in {429, 503}:
        return None
    try:
        seconds = int(retry_after)
    except (TypeError, ValueError):
        return 60
    return max(1, min(seconds, cap))

for status, header in [(200, None), (429, "120"), (503, "soon")]:
    print(status, retry_delay(status, header))

A bounded wait, not an infinite sleep

Try it yourself

Change the values, predict the output, then run the code. The Python runtime is downloaded the first time you press Run, so the first run can take a few seconds.