Validation Is Not a Formality

In epidemiology, we don't trust an intervention because it worked in the trial that funded it. We trust it because it kept working when someone else, with a different population and no stake in the outcome, tried to break it. Clinical AI has largely skipped that second step.

The pattern is familiar to anyone who's reviewed enough validation studies: a model trained and tested on data from the same handful of academic medical centers, reporting strong performance, and then deployed broadly on the implicit assumption that strong performance travels. It often doesn't. Population composition, documentation habits, even the distribution of disease severity differ enough between institutions that a model's accuracy can degrade in ways nobody notices until an audit — if there's ever an audit.

What makes this hard to fix isn't technical. We know how to design a proper external validation study. It's that the incentive structure around clinical AI rewards speed to market far more than it rewards the kind of study whose entire purpose is finding out where the product breaks. Nobody wants to fund the study designed to embarrass their own system.

I don't think the answer is slower innovation for its own sake. I think it's treating external validation the way we already treat a drug trial's phase separation — not a courtesy, a structural requirement before broad claims get made. A model's performance on the population it trained on tells you almost nothing about how it will behave on the population it hasn't met yet.

The systems I trust most are the ones whose creators seem uncomfortable, not proud, when asked about their validation data. Discomfort is usually a sign someone has actually looked.