Python tokens and text vectors Exercises With Answers

What does tokenization do?

1. Deletes rare words
2. Encrypts the text
3. Trains a neural network
4. Splits text into smaller unitscorrect

Answer: Splits text into smaller units

Why: Tokenization turns a string into tokens that later steps can index or embed.

Why build one shared vocabulary?

1. So every document uses the same vector dimensionscorrect
2. To remove punctuation from Python code
3. To guarantee semantic understanding
4. To make all sentences the same length

Answer: So every document uses the same vector dimensions

Why: A shared vocabulary gives each token a stable position across every document.

What does a bag-of-words vector ignore?

1. The vocabulary
2. Word ordercorrect
3. The vector length
4. Token counts

Answer: Word order

Why: Bag-of-words keeps counts but discards the order in which tokens appeared.

What is the output?

import re
def tokenize(text):
    return re.findall(r"[a-z']+", text.lower())
print(tokenize("Python, text!"))

1. ['Python', 'text']
2. ['python', 'text']correct
3. ['python text']
4. ['python,', 'text!']

Answer: ['python', 'text']

Why: The text is lowercased and the regex keeps letters only, so punctuation is removed.

Complete the regular expression that extracts lowercase word tokens.

import re
text = "Python, text!"
print(re.findall(r"____", text.lower()))

Answer: [a-z']+

Why: The character class accepts letters and apostrophes; the plus sign allows one or more.

Complete the pairing function used by the dot product.

a = [1, 2]
b = [3, 4]
print(sum(x * y for x, y in ____(a, b)))

Answer: zip

Why: zip pairs corresponding elements from the two vectors.

Complete the zero-vector guard.

import math

def cosine(a, b):
    norm_a = math.sqrt(sum(x * x for x in a))
    norm_b = math.sqrt(sum(y * y for y in b))
    if not norm_a or not ____:
        return 0.0
    return 1.0

print(cosine([0, 0], [1, 1]))

Answer: norm_b

Why: Either zero length would make the denominator zero, so both must be checked.

Complete the vocabulary lookup used to increment a count.

index = {"python": 0, "text": 1}
counts = [0, 0]
counts[____["text"]] += 1
print(counts)

Answer: index

Why: The index maps a token to its position in the vector.

What does this code print?

import re

def tokenize(text):
    return re.findall(r"[a-z']+", text.lower())

print(tokenize("Python makes text small."))

Answer: ['python', 'makes', 'text', 'small']

Why: The function lowercases the sentence and extracts four word tokens.

What does the program output?

def vector(text, index):
    counts = [0] * len(index)
    for word in text.split():
        if word in index:
            counts[index[word]] += 1
    return counts

index = {"python": 0, "text": 1}
print(vector("python text text", index))

Answer: [1, 2]

Why: The vector records one python token and two text tokens.

What appears on the screen?

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    norm_a = math.sqrt(sum(x * x for x in a))
    norm_b = math.sqrt(sum(y * y for y in b))
    return dot / (norm_a * norm_b)

print(round(cosine([1, 1], [2, 2]), 1))

Answer: 1.0

Why: The vectors point in the same direction, so cosine similarity is 1.0.

What is printed?

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    norm_a = math.sqrt(sum(x * x for x in a))
    norm_b = math.sqrt(sum(y * y for y in b))
    if not norm_a or not norm_b:
        return 0.0
    return dot / (norm_a * norm_b)

print(round(cosine([1, 0], [0, 1]), 1))

Answer: 0.0

Why: The vectors share no dimensions, so the dot product is zero.