The Difference Between a Tool and a Colleague

The first decision-support alert I ever silenced was, in retrospect, correct. It flagged a deteriorating trend in a patient's lactate that I'd been tracking manually and had, if I'm honest, already half-decided to escalate. The alert didn't tell me anything I didn't know. What it also didn't tell me was why it had fired at that particular threshold, on that particular patient, with that particular history — so I dismissed it the way I'd dismiss a colleague who gave me an answer without their reasoning. Not because they were wrong. Because I had no way to build on what they'd told me.

That instinct — to withhold trust from a conclusion you can't interrogate — isn't a failure of clinical culture. It's the correct posture for anyone whose decisions have consequences that don't reset at the end of the day. And yet a great deal of clinical software is still built by people who treat explainability as a regulatory hurdle to clear rather than the actual point of the product. You cannot retrofit trust onto a system that was designed to be believed rather than understood.

What changed my thinking wasn't a single elegant interface. It was watching a nurse on my unit, three years into using a redesigned deterioration-monitoring system, start disagreeing with it out loud, in real time, in front of the family — and being right roughly a third of the time. That ratio is not a failure rate. It's evidence that the system had become legible enough to argue with, which is a categorically different achievement than being accurate enough to obey.

I don't think the future of clinical software looks like fewer overrides. I think it looks like overrides that are traceable, discussable, and occasionally instructive to the system itself — because the alternative, a tool clinicians either blindly follow or quietly route around, has never once produced better outcomes than a colleague you can push back on.

None of this is an argument against automation, confidence scores, or aggressive early-warning thresholds. It's an argument that the reasoning has to ship with the recommendation, not live in a validation study nobody at the bedside will ever read. A clinician who understands why a system is worried will act on that worry differently — and more correctly — than one who's just been told to be worried.