Python model evaluation and accuracy

A model that predicts well on the data it memorized has not proved anything. Evaluation needs a separate test set, a clear positive class, and metrics that match the cost of mistakes. Accuracy is a starting point, not the whole report. You will also combine precision and recall with F1 and inspect subgroup results.

Hold out a test set

Split examples before fitting or tuning anything. The test set should not influence feature selection, scaling, or threshold choice. For small datasets, use several folds and report the variation instead of trusting one lucky split.

rows = [("a", 1), ("b", 0), ("c", 1), ("d", 0), ("e", 1)]
cut = 3
train = rows[:cut]
test = rows[cut:]
print(len(train), len(test))
print(test)

A visible, fixed split

Count the four outcomes

For a binary classifier, every prediction falls into true positive, true negative, false positive, or false negative. Build the confusion matrix explicitly. The names depend on which class you call positive, so state that choice in the report.

actual = [1, 1, 0, 0]
predicted = [1, 0, 0, 1]
tp = sum(a == 1 and p == 1 for a, p in zip(actual, predicted))
tn = sum(a == 0 and p == 0 for a, p in zip(actual, predicted))
fp = sum(a == 0 and p == 1 for a, p in zip(actual, predicted))
fn = sum(a == 1 and p == 0 for a, p in zip(actual, predicted))
print(tp, tn, fp, fn)

Four counts, not one

Compute precision and recall

Precision asks how many predicted positives were correct. Recall asks how many real positives were found. Both can be gamed by a degenerate classifier, so report them together with the class balance and the decision threshold.

tp, fp, fn = 8, 2, 4
precision = tp / (tp + fp)
recall = tp / (tp + fn)
print(round(precision, 2))
print(round(recall, 2))

Two different questions

Beware accuracy on imbalance

If only five of one hundred examples are positive, predicting negative every time gives 95% accuracy and finds none of the positives. Always compare against a trivial baseline and inspect per-class results before celebrating a headline number.

actual = [0] * 95 + [1] * 5
predicted = [0] * 100
accuracy = sum(a == p for a, p in zip(actual, predicted)) / len(actual)
print(round(accuracy, 2))
print(sum(1 for a, p in zip(actual, predicted) if a == 1 and p == 1))

High accuracy, zero recall

F1 balances precision and recall

F1 is the harmonic mean of precision and recall. It is high only when both are high, so it is useful when neither false positives nor false negatives can be ignored. Report the component metrics too; F1 hides which side is failing.

tp, fp, fn = 8, 2, 4
precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * precision * recall / (precision + recall)
print(round(precision, 2), round(recall, 2), round(f1, 2))

One number, two ingredients

Check slices, not only the overall score

An aggregate can look acceptable while one group fails badly. Group results by a relevant slice, but keep groups large enough to avoid overreading noise. Slicing is a diagnostic, not permission to collect unnecessary personal data.

rows = [
    ("new", 1, 1), ("new", 1, 0), ("new", 0, 1),
    ("returning", 1, 1), ("returning", 0, 0), ("returning", 0, 0),
]
for group in sorted({row[0] for row in rows}):
    subset = [row for row in rows if row[0] == group]
    correct = sum(actual == predicted for _, actual, predicted in subset)
    print(group, correct, "/", len(subset), round(correct / len(subset), 2))

A slice can hide inside a good average

Try it yourself

Change the values, predict the output, then run the code. The Python runtime is downloaded the first time you press Run, so the first run can take a few seconds.