Python Naive Bayes Exercises: Counts, Priors, Smoothing
A Naive Bayes classifier is mostly careful counting. These exercises build the count tables, turn them into smoothed probabilities, compare classes in log space, and check the result on text that was not used for training.
Exercise: build one count table per class
The label is not a feature. Split the training documents by label, then count each token inside each group. Keep the tokenization rule identical for every document.
docs = [("good movie", "pos"), ("great film", "pos"), ("bad movie", "neg"), ("boring film", "neg")]
counts = {}
for text, label in docs:
counts.setdefault(label, {})
for word in text.split():
counts[label][word] = counts[label].get(word, 0) + 1
print(counts["pos"])
print(counts["neg"])Evidence grouped by label
Exercise: compute the class priors
The prior is the fraction of documents in a class. It matters when one class is much more common. Print both priors and check that they sum to one.
labels = ["pos", "pos", "neg", "neg"]
for label in ("pos", "neg"):
print(label, labels.count(label) / len(labels))How common was each class?
Exercise: smooth an unseen word
Without smoothing, a word that never appeared in a class would make the whole product zero. Add one to every count, and add the vocabulary size to the denominator.
counts = {"great": 0}
total = 4
vocab_size = 6
probability = (counts.get("great", 0) + 1) / (total + vocab_size)
print(round(probability, 2))A small probability, not zero
Exercise: compare log scores
The class score is the log prior plus one log likelihood per token. Compare two candidate scores without multiplying tiny numbers. The larger score wins; keep a deterministic tie rule.
import math
pos = math.log(0.5) + math.log(0.2) + math.log(0.2)
neg = math.log(0.5) + math.log(0.1) + math.log(0.1)
print(round(pos, 2), round(neg, 2))
print("pos" if pos > neg else "neg")Add logs, then compare
Exercise: predict held-out text
Build the model from training documents, then classify new documents. Accuracy on two examples is a smoke test, not evidence of production quality.
import math
train = [("good movie", "pos"), ("great film", "pos"), ("bad movie", "neg"), ("boring film", "neg")]
tests = [("great movie", "pos"), ("boring movie", "neg")]
vocab = sorted({word for text, _ in train for word in text.split()})
counts = {}
priors = {}
for text, label in train:
priors[label] = priors.get(label, 0) + 1
counts.setdefault(label, {})
for word in text.split():
counts[label][word] = counts[label].get(word, 0) + 1
def predict(text):
scores = {}
for label in counts:
total = sum(counts[label].values())
score = math.log(priors[label] / len(train))
for word in text.split():
score += math.log((counts[label].get(word, 0) + 1) / (total + len(vocab)))
scores[label] = score
return max(scores, key=scores.get)
correct = sum(predict(text) == label for text, label in tests)
print(correct / len(tests))A smoke test, not a benchmark
Your practice checklist
- Use the same tokenizer for training and prediction.
- Keep the label out of the feature vocabulary.
- Add Laplace smoothing to every vocabulary word.
- Score in log space and define a tie rule.
- Evaluate on held-out text and report the sample size.
Common questions
Does Naive Bayes require a neural network?
No. It is a counting-based probabilistic classifier and a strong baseline for many text tasks.
What if a word is not in the vocabulary?
The usual policy is to ignore it or map it to an unknown-token feature that was present during training. Decide before evaluation.
Why are all my log scores negative?
Probabilities are at most one, so their logarithms are at most zero. Compare relative values, not absolute sign.
Practice counting and scoring
12 questions on counts, priors, smoothing, log scores, and held-out predictions.
Keep reading: