Classifying customers by purchase value: compare models before choosing
On 133 customers never seen during training, 109 correct answers, that is 82%. Reading the error matrix tells you which model to keep.
Two groups of customers
The project classified customers into two groups: those whose purchase value is above 300, and the others. Knowing in advance which group a customer falls into makes it possible to offer the right thing at the right time.
Judging on customers never seen
A model is measured on data it did not see during training. Here, 133 test customers: 109 are classified correctly, 71 in the more modest group and 38 in the group above 300.
We always compare candidate models on the same test data, with the same rules. Otherwise you are comparing luck, not models.
An overall score separates two models poorly; their errors separate them well.
Not all errors are equal
The matrix shows the 24 errors. Fifteen customers were placed above 300 when they are not; nine customers above 300 were missed.
Depending on the use, the costs differ. An offer sent by mistake costs little; a forgotten high-value customer can cost a lot. Between two close models, we keep the one that makes the costliest error least often.
Key points
- Measure on unseen data, the same for every model.
- Read the error matrix, not just the percentage.
- Choose the model by the error that costs the business most.