Vertical AI Wins Where It Matters

The technology industry tells a comfortable story: large language models now pass medical licensing exams at physician-level scores. If a model can pass the test that gates entry to the profession, surely it is ready to do the work.

The evidence says otherwise. A systematic review of thirty-nine medical benchmarks found a knowledge-practice gap of thirty-nine to forty-five percentage points, roughly the difference between an A and an F. On multiple-choice knowledge tests, leading models scored eighty-four to ninety percent. On benchmarks designed to imitate actual clinical practice, performance fell to forty-five to sixty-nine percent. Safety tasks, recognising dangerous drug interactions, catching contraindications, sat at just forty to fifty percent. The weakest category is the one where errors are counted in lives, not marks.

A second study tested two thousand and sixty-nine open-ended clinical scenarios across twenty-six specialties. Six leading models averaged 57.2 percent overall, and every one declined by a statistically significant 13.3 percent precisely when stakes were highest. Examination performance is not a proxy for clinical readiness. It is a misleading one.

Domain-tuned models do not win everywhere, and the honest picture matters. On open-ended reasoning and broad recall, frontier general models are strong and frequently ahead. The accuracy gap that once flattered specialised systems has largely closed on the tasks leaderboards measure.

But on the bounded, document-grounded work that fills the actual working day of a clinician or analyst, a well-built vertical system holds a real and repeatable edge. In a blinded study, practising physicians preferred a healthcare-specific model over a vastly larger general system by roughly two to one on clinical documentation tasks. An eight-billion-parameter model, small enough to run on a hospital’s own hardware, outperformed a frontier system on the work that actually matters at the bedside.

Finance confirms the pattern. A domain-tuned finance model beat a leading general model on the routine classification tasks that fill the regulated working day. A separate study of financial embedding models found that general benchmark scores were a poor predictor of performance on financial retrieval, where words like “liability” carry weight ordinary language never assigns them.

Capability follows construction, not category. What separates a domain model that beats a frontier system from one that loses is not the label but the quality of the engineering underneath.

Once raw accuracy is granted to be roughly a draw on leaderboard tasks, the question of which system to trust inside a hospital or a bank stops being about scores. It becomes a question about control, traceability, and data governance, dimensions that do not appear on any benchmark.

A model an institution can hold in its own hands, retrain on its own data, and audit end-to-end answers questions a closed frontier model accessed over an API simply cannot. A black box on a third party’s infrastructure cannot be frozen to a validated version, cannot be retrained on confidential data without sending that data outside, and cannot be fully interrogated when something goes wrong.

In India, these are no longer preferences, they are increasingly the text of the law. The Digital Personal Data Protection Act, 2023 converts traceability, control, and accountability into enforceable obligations carrying penalties up to two hundred and fifty crore rupees. The CDSCO’s draft Guidance on Medical Device Software, consultative, not yet binding, places clinical AI under device regulation, demanding version control, architecture documentation, and a defined protocol for retraining and rollback. Two instruments, approaching from opposite directions, arrive at the same requirement: a deployed AI system must be traceable, controllable, and anchored to an accountable party within reach of the regulator. Domain-built systems meet these requirements by design. General systems retrofitted for compliance meet them, if at all, by effort.