This answer key contains 4 multiple-choice, 4 fill-in-the-blank, and 4 output-prediction questions for Python robots.txt and polite scraping, with the exact answer and a short explanation for every question.
The answers are shown in full, so use this page after making an attempt. The practice page is the better first pass.
What does Disallow: /private mean?
Answer: Matching crawlers should not fetch paths under /private
Why: Disallow publishes a crawl preference for matching user agents.
Where is robots.txt normally published?
Answer: At the site root, /robots.txt
Why: Crawlers look for /robots.txt at the root of the site.
What should a crawler do after a 429 response?
Answer: Slow down and retry later
Why: 429 means too many requests; back off and respect Retry-After when provided.
Which action is appropriate for a public page?
Answer: Collect only the needed public fields within site terms
Why: Public reachability does not remove terms, privacy, or crawl limits.
Complete the directive that names the crawler group.
line = "____: *"
key = line.split(":", 1)[0]
print(key)Answer: User-agent
Why: User-agent begins a group of rules for a named client.
Complete the directive that blocks a path.
robots = "User-agent: *\nDisallow: /private"
for line in robots.splitlines():
if line.startswith("____"):
print(line)Answer: Disallow
Why: Disallow marks paths that matching crawlers should not fetch.
Complete the directive that sets a delay between requests.
key = "____"
print(key in {"User-agent", "Disallow", "Crawl-delay", "Sitemap"})Answer: Crawl-delay
Why: Crawl-delay suggests a pause between requests for a crawler group.
Complete the directive that points to a sitemap.
line = "____: https://example.com/sitemap.xml"
print(line.startswith("Sitemap"))Answer: Sitemap
Why: Sitemap gives crawlers a URL for the site's sitemap.
What does this code print?
robots = "User-agent: *\nDisallow: /private\nCrawl-delay: 2"
rules = {}
for line in robots.splitlines():
if ":" in line:
key, value = line.split(":", 1)
rules[key.strip().lower()] = value.strip()
print(rules["disallow"])Answer: /private
Why: The Disallow value is /private.
What does the program output?
robots = "User-agent: *\nDisallow: /private\nCrawl-delay: 2"
rules = {}
for line in robots.splitlines():
if ":" in line:
key, value = line.split(":", 1)
rules[key.strip().lower()] = value.strip()
print(rules["crawl-delay"])Answer: 2
Why: The Crawl-delay value is the string 2.
What appears on the screen?
rules = [("allow", "/docs"), ("disallow", "/private")]
path = "/private/report"
decision = "allowed"
for kind, prefix in rules:
if path.startswith(prefix):
decision = kind
print(decision)Answer: disallow
Why: The path matches the longer /private rule, so the decision becomes disallow.
What is printed?
record = {"name": "Ada", "email": "[email protected]", "bio": "Python"}
public = {key: value for key, value in record.items() if key != "email"}
print(public)Answer: {'name': 'Ada', 'bio': 'Python'}
Why: The dictionary comprehension removes the email field.
Try the questions first
Open the practice page for Python robots.txt and polite scraping and answer the same 12 questions with instant feedback.