Python Pagination Exercises: Queues, Visits, and Limits

Use a FIFO queue

A deque with append and popleft gives breadth-first order. That is usually what you want for pagination: finish the current listing level before following detail pages. A plain list with pop(0) is quadratic for large frontiers.

from collections import deque

queue = deque(["/page/1"])
seen = []
while queue:
    url = queue.popleft()
    seen.append(url)
    if url == "/page/1":
        queue.extend(["/page/2", "/page/3"])
print(seen)

First in, first out

Mark a page when you enqueue it

If you wait until a page is processed to mark it visited, two parents can enqueue the same child. Add the URL to visited at enqueue time, and check before every append. The set and the queue then agree on what is pending.

queue = ["/a"]
visited = {"/a"}
links = {"/a": ["/b", "/b", "/c"], "/b": ["/a"]}
order = []
while queue:
    page = queue.pop(0)
    order.append(page)
    for link in links.get(page, []):
        if link not in visited:
            visited.add(link)
            queue.append(link)
print(order)

No duplicate work

Spend a page budget

A budget makes the crawl reproducible and prevents an accidental infinite run. Decrement before parsing or after a successful fetch, but document which one. Stop cleanly when the budget reaches zero and report that the frontier is unfinished.

queue = ["/1", "/2", "/3", "/4"]
budget = 2
done = []
while queue and budget:
    done.append(queue.pop(0))
    budget -= 1
print(done)
print("remaining", len(queue))

A finite run

Keep the next link explicit

Many sites paginate with a rel=next link rather than a predictable number. Parse that link and resolve it against the current page. Do not guess /page/2 from /page/1 unless the site documents that pattern.

from html.parser import HTMLParser
from urllib.parse import urljoin

class Next(HTMLParser):
    def __init__(self):
        super().__init__()
        self.href = None
    def handle_starttag(self, tag, attrs):
        values = dict(attrs)
        if tag == "a" and values.get("rel") == "next" and self.href is None:
            self.href = values.get("href")

p = Next()
p.feed('<a rel="next" href="/page/2">Next</a>')
print(urljoin("https://example.com/page/1", p.href))

Follow the declared next link

Make output order deterministic

Sets are useful for membership, not for stable output. Convert visited URLs to a sorted list when writing a report, or keep a separate insertion-ordered list. Deterministic output makes diffs and tests meaningful.

visited = {"/b", "/a", "/c"}
print(sorted(visited))
print(list(dict.fromkeys(["/b", "/a", "/b", "/c"])))

Sort before reporting

Your practice checklist

• Use deque.popleft for a breadth-first frontier.
• Add a URL to visited when it enters the queue.
• Set and report a page or request budget.
• Resolve rel=next against the current page.
• Sort or preserve insertion order for stable output.

Common questions

Keep reading:

• Python pagination and deduplication
• Python Exercises With Answers: How to Practice Without Copying
• Python Practice for Beginners: A Simple 4-Week Plan