Python Tokenization Exercises: Tokens, Vectors, Cosine

Normalize before you split

Lowercasing is a simple normalization step. It makes Python and python the same token, but it also erases information that may matter for names or code. Choose a rule and apply it consistently to training and new text.

import re

def tokenize(text):
    return re.findall(r"[a-z']+", text.lower())

print(tokenize("Python, Python! Text."))

One rule for every document

Share one vocabulary

Build the vocabulary from the training documents, not separately for each new document. Sort the tokens so the dimension order is deterministic. Tokens outside the vocabulary need an explicit policy, such as ignore or unknown.

docs = ["python text", "text models"]
vocab = sorted({word for doc in docs for word in doc.split()})
index = {word: i for i, word in enumerate(vocab)}
print(index)

Stable dimensions

Count into vectors

A count vector is easy to inspect: each position says how many times one vocabulary token appeared. Normalize by document length when long documents should not automatically look more similar to everything.

def vector(text, index):
    counts = [0] * len(index)
    for word in text.split():
        if word in index:
            counts[index[word]] += 1
    return counts

index = {"python": 0, "text": 1}
print(vector("python text text", index))

A vector you can audit

Compare with cosine

Cosine similarity measures the angle between vectors, so it ignores magnitude. That is useful when one document is simply longer. It still cannot tell you whether two sentences mean the same thing if they use different words.

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    norm_a = math.sqrt(sum(x * x for x in a))
    norm_b = math.sqrt(sum(y * y for y in b))
    if not norm_a or not norm_b:
        return 0.0
    return dot / (norm_a * norm_b)

print(round(cosine([1, 1], [2, 2]), 3))
print(round(cosine([1, 0], [0, 1]), 3))

Direction, not length

Know what the bag misses

A bag-of-words model gives 'dog bites man' and 'man bites dog' the same vector. Character n-grams can help with misspellings, and word embeddings can capture distributional similarity, but neither removes the need to evaluate on your actual task.

def bag(text):
    return sorted(text.lower().split())

print(bag("dog bites man") == bag("man bites dog"))

Order is gone

Your practice checklist

• Use the same normalization and tokenizer for training and inference.
• Build one vocabulary and keep its order stable.
• Decide explicitly how to handle unknown tokens.
• Guard cosine similarity against zero-length vectors.
• Test the pipeline on paraphrases, misspellings, and reordered sentences.

Common questions

Keep reading:

• Python tokens and text vectors
• Python Exercises With Answers: How to Practice Without Copying
• Python Practice for Beginners: A Simple 4-Week Plan