Python tokens and text vectors

AI systems do not read sentences the way people do. They start by splitting text into tokens and representing those tokens as numbers. This lesson builds a small bag-of-words vector and compares documents with cosine similarity, so the arithmetic behind the idea is visible. You will also handle typos with character n-grams and downweight common words with IDF.

Normalize, then tokenize

Lowercasing reduces accidental differences, and a regular expression extracts word-like tokens. Real tokenizers may split punctuation, subwords, or Unicode in different ways. The important habit is to make the tokenization rule explicit and reuse it everywhere.

import re

def tokenize(text):
    return re.findall(r"[a-z']+", text.lower())

print(tokenize("Python makes text small."))

A clear, repeatable rule

Build a shared vocabulary

A vector needs a fixed order of dimensions. Collect the unique tokens from the training documents, sort them for determinism, and map each token to an index. A token outside this vocabulary is usually ignored or mapped to an unknown-token slot.

docs = ["python text", "text models"]
vocab = sorted({word for doc in docs for word in doc.split()})
index = {word: i for i, word in enumerate(vocab)}
print(vocab)
print(index["text"])

Vocabulary, then index

Count tokens into a vector

A bag-of-words vector records how often each vocabulary token appears. It deliberately ignores word order. That makes it fast and easy to inspect, but it also means 'dog bites man' and 'man bites dog' receive the same vector.

def vector(text, index):
    counts = [0] * len(index)
    for word in text.split():
        if word in index:
            counts[index[word]] += 1
    return counts

index = {"python": 0, "text": 1}
print(vector("python text text", index))

Counts, not grammar

Compare with cosine similarity

The dot product rewards shared tokens, while the vector lengths keep long documents from dominating. Cosine similarity is the dot product divided by both lengths. Check for a zero-length vector before dividing, and remember that lexical overlap is not the same as meaning.

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    norm_a = math.sqrt(sum(x * x for x in a))
    norm_b = math.sqrt(sum(y * y for y in b))
    if not norm_a or not norm_b:
        return 0.0
    return dot / (norm_a * norm_b)

print(round(cosine([1, 1], [1, 0]), 3))
print(round(cosine([1, 0], [0, 1]), 3))

Shared terms raise the score

Character n-grams survive typos and unseen words

A word-level vocabulary treats a misspelling as an unknown token. Character n-grams keep overlapping fragments, so a query can still match a known word approximately. The trade-off is a larger feature space and more false matches; use n-grams when spelling and morphology matter.

def trigrams(text):
    padded = " " + text.lower().replace(" ", "_") + " "
    return {padded[i:i + 3] for i in range(len(padded) - 2)}

query = trigrams("lattee")
known = trigrams("latte")
print(sorted(query & known))
print(round(len(query & known) / len(query | known), 2))

Shared fragments instead of an exact token

Downweight common tokens with IDF

Raw counts make frequent words dominate. Inverse document frequency gives a token less weight when it appears in many documents. A smoothed IDF avoids division by zero, and the resulting weighted vectors can still be compared with cosine similarity.

import math

docs = ["python text models", "python models", "text models"]
vocab = sorted({word for doc in docs for word in doc.split()})
df = {word: sum(word in doc.split() for doc in docs) for word in vocab}
idf = {word: math.log((1 + len(docs)) / (1 + df[word])) + 1 for word in vocab}

def vector(doc):
    words = doc.split()
    return [words.count(word) * idf[word] for word in vocab]

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    na = math.sqrt(sum(x * x for x in a))
    nb = math.sqrt(sum(y * y for y in b))
    return dot / (na * nb) if na and nb else 0.0

print({word: round(idf[word], 2) for word in vocab})
print([round(x, 2) for x in vector("python text")])
print(round(cosine(vector("python text"), vector("text models")), 2))

Rare terms carry more of the signal

Try it yourself

Change the values, predict the output, then run the code. The Python runtime is downloaded the first time you press Run, so the first run can take a few seconds.