Python robots.txt and polite scraping

A working parser is not permission to collect anything you can reach. robots.txt publishes crawl preferences, site terms set additional rules, and privacy law may restrict personal data. Politeness is technical: identify yourself, limit requests, cache, and stop when the site says no. You will also let RobotFileParser apply path rules and parse retry timing.

Read the published rules

robots.txt is a plain-text file at the site root. It contains groups for user agents and directives such as Allow, Disallow, Crawl-delay, and Sitemap. It is a crawl preference, not a license or a substitute for the site's terms.

robots = "User-agent: *\nDisallow: /private\nAllow: /docs\nCrawl-delay: 2"
rules = {}
for line in robots.splitlines():
    if ":" in line:
        key, value = line.split(":", 1)
        rules[key.strip().lower()] = value.strip()
print(rules["disallow"])
print(rules["crawl-delay"])

A tiny key-value view

Respect the matching user-agent group

Rules are grouped by User-agent. A group for * applies when no more specific group matches. Production crawlers should use a maintained robots parser; the small function here shows the shape of the problem, not the full standard.

def rules_for(text, agent):
    active = False
    rules = []
    for line in text.splitlines():
        line = line.strip()
        if not line or line.startswith("#"):
            continue
        key, _, value = line.partition(":")
        key = key.strip().lower()
        value = value.strip()
        if key == "user-agent":
            active = value == "*" or value.lower() == agent.lower()
        elif active and key in {"allow", "disallow"}:
            rules.append((key, value))
    return rules

text = "\n".join([
    "User-agent: *",
    "Disallow: /private",
    "User-agent: GoodBot",
    "Allow: /private/public",
])
print(rules_for(text, "GoodBot"))

Groups, not a flat dictionary

Check the path before the request

A path check should happen before the URL enters the queue. Longer and more specific rules usually win, and Allow can override a broader Disallow. Treat your local implementation as a guardrail, not a legal opinion.

rules = [("allow", "/docs"), ("disallow", "/private")]
path = "/private/report"
decision = "allowed"
for kind, prefix in rules:
    if path.startswith(prefix):
        decision = kind
print(decision)

A simple path guard

Slow down and minimize data

Respect Crawl-delay and Retry-After, back off on 429 or 503, and cache pages you already have. Collect only the fields the task requires. Do not bypass logins, paywalls, CAPTCHAs, or access controls, and do not collect personal data without a lawful basis.

record = {"name": "Ada", "email": "[email protected]", "bio": "Python"}
public = {key: value for key, value in record.items() if key != "email"}
print(public)

Keep only what the task needs

Let RobotFileParser judge path rules

Do not implement robots.txt precedence with a single substring check. urllib.robotparser handles groups, wildcard agents, and longer Allow/Disallow paths. Parse a captured file in tests, then ask can_fetch() for the exact user agent and path.

from urllib.robotparser import RobotFileParser

lines = [
    "User-agent: *",
    "Disallow: /private",
    "Allow: /private/public",
    "",
    "User-agent: BadBot",
    "Disallow: /",
]
robots = RobotFileParser()
robots.parse(lines)
for path in ["/", "/private", "/private/public"]:
    print(path, robots.can_fetch("PyPracticeBot", path))
print(robots.can_fetch("BadBot", "/"))

Specific groups and longer paths

Parse Retry-After and Crawl-delay

Retry-After can be a number of seconds or an HTTP date. Crawl-delay is a request spacing hint, not a promise that the site can handle your concurrency. Convert both to a bounded wait, and fall back to a conservative default when the value is missing or malformed.

from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

def retry_seconds(value, now):
    try:
        return max(0, int(value))
    except (TypeError, ValueError):
        try:
            when = parsedate_to_datetime(value)
        except (TypeError, ValueError):
            return 60
        return max(0, int((when - now).total_seconds()))

now = datetime(2026, 10, 28, 12, 0, tzinfo=timezone.utc)
for raw in ["120", "Wed, 28 Oct 2026 12:02:00 GMT", "bad"]:
    print(raw, retry_seconds(raw, now))
print("Crawl-delay", 2)

Seconds, dates, and a fallback

Try it yourself

Change the values, predict the output, then run the code. The Python runtime is downloaded the first time you press Run, so the first run can take a few seconds.