Complaints and compliments use different words, and counting them is enough for a first classifier. Naive Bayes assumes an independence it does not have. Why does it still work?
Naive Bayes is a fast text-classification baseline built from counting. It assumes words are independent given the class, which is usually false, yet the model is still useful. The lesson makes the counts, priors, smoothing, and log-space scoring explicit. You will also see why smoothing matters for unseen words and why longer documents get more negative scores.
Count tokens by class
Training starts with labeled documents. For each class, count how often every token appears. These counts are the raw evidence the classifier will use; the label is never included as a feature.
docs = [
("good movie", "pos"),
("great film", "pos"),
("bad movie", "neg"),
("boring film", "neg"),
]
counts = {}
for text, label in docs:
counts.setdefault(label, {})
for word in text.split():
counts[label][word] = counts[label].get(word, 0) + 1
print(counts["pos"]["movie"])
print(counts["neg"]["bad"])Every class gets its own count table
Priors and smoothed likelihoods
The prior is the share of training documents in a class. The likelihood of a word is its count in that class plus one, divided by the class total plus the vocabulary size. Laplace smoothing prevents an unseen word from making a probability zero.
docs = [
("good movie", "pos"),
("great film", "pos"),
("bad movie", "neg"),
("boring film", "neg"),
]
vocab = sorted({word for text, _ in docs for word in text.split()})
counts = {}
for text, label in docs:
counts.setdefault(label, {})
for word in text.split():
counts[label][word] = counts[label].get(word, 0) + 1
for label in sorted(counts):
total = sum(counts[label].values())
probability = (counts[label].get("great", 0) + 1) / (total + len(vocab))
print(label, round(probability, 3))Add one count, then normalize
Score in log space
Multiplying many small probabilities can underflow to zero. Adding logarithms is numerically safer and preserves the ranking. A log probability is negative, and a larger value means a more likely class.
import math
print(round(math.log(0.5 * 0.2), 3))
print(round(math.log(0.5) + math.log(0.2), 3))Same ranking, safer arithmetic
Predict the most likely class
For a new document, add the log prior to the smoothed log likelihood of each token. Choose the class with the larger score. Keep a tie rule, and evaluate the classifier on documents it did not train on.
import math
docs = [
("good movie", "pos"),
("great film", "pos"),
("bad movie", "neg"),
("boring film", "neg"),
]
vocab = sorted({word for text, _ in docs for word in text.split()})
counts = {}
class_docs = {}
for text, label in docs:
class_docs[label] = class_docs.get(label, 0) + 1
counts.setdefault(label, {})
for word in text.split():
counts[label][word] = counts[label].get(word, 0) + 1
def predict(text):
scores = {}
for label in counts:
total = sum(counts[label].values())
score = math.log(class_docs[label] / len(docs))
for word in text.split():
score += math.log((counts[label].get(word, 0) + 1) / (total + len(vocab)))
scores[label] = score
return max(scores, key=scores.get)
print(predict("great film"))The class with the larger log score
Smoothing keeps an unseen word finite
With no smoothing, an unseen word has likelihood zero, and its logarithm is undefined. Add-alpha smoothing gives every vocabulary word a small nonzero count. Alpha is a choice: too little can still be brittle, too much washes out the evidence.
import math
count = 0
total = 2
vocab_size = 3
for alpha in [0, 1]:
probability = (count + alpha) / (total + alpha * vocab_size)
print(alpha, probability, round(math.log(probability), 3) if probability else "undefined")A zero count needs a policy
Longer documents get more negative scores
A Naive Bayes score adds one log likelihood per token. A longer document therefore tends to have a more negative total score even when its average evidence is similar. Compare average log likelihood or calibrate on validation data before treating scores as probabilities.
import math
log_prior = math.log(0.5)
log_word = math.log(0.8)
def score(tokens):
return log_prior + len(tokens) * log_word
for tokens in [["good"], ["good", "good"], ["good", "good", "good"]]:
print(len(tokens), round(score(tokens), 2), round((score(tokens) - log_prior) / len(tokens), 2))Total score versus per-token score
Key points
- Naive Bayes learns from token counts grouped by class.
- The prior reflects how common each class was in training.
- Laplace smoothing keeps unseen words from zeroing a score.
- Adding logs avoids underflow when multiplying probabilities.
- The independence assumption is a simplifying choice, not a fact.
- Smoothing handles unseen words; document length changes the scale of a log score.
Common mistakes
- Including the label itself as a token feature.
- Smoothing only the words that happen to appear in the new document.
- Reporting accuracy on the same documents used to build the counts.
- Comparing raw Naive Bayes scores across documents of very different lengths.
Using the four training documents, compute the smoothed probability of 'bad' given class neg and print it rounded to three decimals.
Try it yourself
Change the values, predict the output, then run the code. The Python runtime is downloaded the first time you press Run, so the first run can take a few seconds.
Practice this lesson
12 questions with immediate explanations.