Polypharmacy: why a good success rate is not enough

The question

The more medicines a person takes at once, the more likely interactions and side effects become. The project set out to estimate, for each person, the risk of being hospitalised from what their records contain.

Volume helps: 9,066,106 records are enough for the model to see rare situations. But volume alone says nothing about the quality of a model.

The success-rate trap

In this kind of data, hospitalised people are a minority. A model answering “not hospitalised” for everyone would have an excellent success rate, while missing every person we are trying to protect.

That is why we never judge such a model on a single percentage. We look separately at what it does for hospitalised people and for the others.

A model is judged on the people it must not miss, not on its average.

Looking at both populations

The project chart shows, for each group, the probability the model gives each record. Non-hospitalised people gather at low probabilities, hospitalised people at high ones: the model separates the two well.

Errors sit where the two groups overlap, around the threshold. That is where to look closely, and where the decision is made.

The threshold is set with those who decide

A 0.5 threshold is a default setting, not a medical decision. In a hospital, missing an at-risk person costs more than one alert too many. The threshold is therefore set with clinicians, according to what each error costs, and it can differ from one ward to another.