Python URL Parsing Exercises: Join, Query, and Clean
Most duplicate links are not really duplicates; they are the same destination written four ways. These exercises turn URL strings into components, resolve them against the page, and rebuild a canonical value.
Read the components
Start by naming the parts. scheme identifies the protocol, netloc the host and port, path the resource, query the parameters, and fragment the in-page anchor. Each part has a different job in a crawl.
from urllib.parse import urlparse
url = urlparse("https://example.com:8080/docs?q=python#intro")
print(url.scheme)
print(url.netloc)
print(url.path)
print(url.query)
print(url.fragment)Name every part
Resolve against the page
A link is relative to the page that contains it. urljoin handles ../, root-relative paths, and query-only references. Joining before storing prevents the queue from treating one destination as several.
from urllib.parse import urljoin
base = "https://example.com/docs/guide/page.html"
for href in ["../index.html", "/about", "?page=2"]:
print(urljoin(base, href))Three relative forms
Read repeated query keys
parse_qs returns lists, even when a key appears once. That is deliberate: query strings can repeat keys. Index the first value only when the API documents a single value, and convert numbers before arithmetic.
from urllib.parse import parse_qs
params = parse_qs("tag=python&tag=web&page=2")
print(params["tag"])
print(int(params["page"][0]) + 1)Lists, then explicit conversion
Encode a new query
urlencode escapes spaces and reserved characters. Use it for search forms and pagination links. Passing a list of pairs preserves repeated keys and order, which a plain dictionary cannot guarantee.
from urllib.parse import urlencode
query = urlencode([("q", "a b"), ("tag", "python"), ("tag", "web")])
print(query)Encode pairs, keep repeats
Remove the fragment before deduplication
The fragment after # is for the browser and is not part of the server request. Strip it before adding a page to the visited set. Keep the original URL if you need to report where a link was found.
from urllib.parse import urlsplit, urlunsplit
parts = urlsplit("https://example.com/docs?q=1#section-2")
clean = urlunsplit((parts.scheme, parts.netloc, parts.path, parts.query, ""))
print(clean)One canonical page URL
Your practice checklist
- Use urlparse instead of slicing URL strings by hand.
- Resolve relative links with urljoin before deduplicating.
- Treat parse_qs values as lists.
- Use urlencode for user input and repeated query keys.
- Strip fragments before comparing page URLs.
Common questions
Does urljoin validate the destination?
No. It resolves the reference syntactically. You still need to check the scheme, host, and crawl rules.
Why does parse_qs return a list?
A query string may contain the same key more than once, so a single value would lose information.
Should I lowercase the whole URL?
No. The host is case-insensitive, but paths and query values may be case-sensitive. Normalize only the parts whose rules you know.
Practice URL parsing
12 questions on urlparse, urljoin, query strings, encoding, and fragments.
Keep reading: