Python Model Evaluation Exercises: Beyond Accuracy

Split before you look

Hold out examples before choosing features or thresholds. A deterministic split is easier to reproduce; a shuffled split with a fixed seed is often better for ordered data. Never tune on the test set.

rows = list(range(10))
cut = 7
train, test = rows[:cut], rows[cut:]
print(len(train), len(test))
print(test)

A fixed holdout

Build the confusion matrix

Count true positives, true negatives, false positives, and false negatives from actual and predicted labels. Print the counts before computing ratios; a surprising metric is often explained by one cell.

actual = [1, 1, 0, 0]
predicted = [1, 0, 0, 1]
tp = sum(a == 1 and p == 1 for a, p in zip(actual, predicted))
tn = sum(a == 0 and p == 0 for a, p in zip(actual, predicted))
fp = sum(a == 0 and p == 1 for a, p in zip(actual, predicted))
fn = sum(a == 1 and p == 0 for a, p in zip(actual, predicted))
print(tp, tn, fp, fn)

Four cells first

Derive precision, recall, and F1

Precision is tp / (tp + fp); recall is tp / (tp + fn). F1 is their harmonic mean. Use a zero-safe helper so a classifier that predicts no positives does not crash the report.

tp, fp, fn = 8, 2, 4
precision = tp / (tp + fp) if tp + fp else 0.0
recall = tp / (tp + fn) if tp + fn else 0.0
f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
print(round(precision, 2), round(recall, 2), round(f1, 2))

Three related numbers

Compare with a trivial baseline

On imbalanced data, always-negative can have high accuracy. Compare your model with that baseline and inspect per-class recall. A model that never predicts the minority class is not useful just because its accuracy is 95%.

actual = [0] * 95 + [1] * 5
predicted = [0] * 100
accuracy = sum(a == p for a, p in zip(actual, predicted)) / len(actual)
print(round(accuracy, 2))
print(sum(1 for a, p in zip(actual, predicted) if a == 1 and p == 1))

Accuracy is not enough

Move the threshold and watch the tradeoff

A classifier score becomes a label only after a threshold. Raising the threshold usually reduces false positives and increases false negatives. Choose the threshold from validation data and the real cost of each error.

scores = [0.9, 0.6, 0.4, 0.3]
for threshold in (0.5, 0.7):
    predicted = [1 if score >= threshold else 0 for score in scores]
    print(threshold, predicted)

A decision, not a fact

Your practice checklist

• Hold out test data before feature selection and tuning.
• Print the confusion matrix before ratios.
• Use zero-safe precision, recall, and F1 calculations.
• Compare against the majority-class baseline.
• Choose thresholds from validation data and error costs.

Common questions

Keep reading:

• Python model evaluation and accuracy
• Python Exercises With Answers: How to Practice Without Copying
• Python Practice for Beginners: A Simple 4-Week Plan