The Doctors Microsoft's AI Beat Were Never in the Room

Eighty-three to thirty-six. That is the scoreline Microsoft brought to Science this week: its MAI-DxO system — an orchestration of AI agents that question a patient, order tests and debate a diagnosis like a panel of doctors — correctly solved complex cases 83 percent of the time in simulation, while generalist physicians working with reference materials managed 36, TechCrunch reports. Mustafa Suleyman, the CEO of Microsoft AI, called the result a step toward "medical superintelligence." He told The Verge the technology is "finally here."

I read a study like this the way I read any invoice: who was measured, on whose scale, and who sends the bill.

The doctors in the comparison were primary care physicians and general internists, working individually. TechCrunch notes the caveat sitting inside the study itself: cases this complex are usually handled by specialists or physician teams — and none of them were measured. Microsoft's machine got to be a committee. The human being measured was not allowed to consult one. The Verge reports the company claims its tool diagnoses "up to four times more accurately" than doctors; that multiplier requires the baseline of physicians working without any references at all, who scored 22 percent. Strip the doctor of books, then of colleagues, and the rout gets louder.

The benchmark those scores come from, SDBench, was compiled by Microsoft itself, as The Verge reports, with a promise to open-source it. Open-sourcing later is welcome. It does not change the fact that the first measurement the world heard was taken on the seller's own ruler, in the same week the company published what TechCrunch calls a manifesto of sorts — one that defines medical superintelligence as the capacity to surpass the strongest contemporary human-led health systems in medical reasoning and coordination across the full arc of care.

The study's own tables, per TechCrunch's reporting, itemize where the gains actually come from. OpenAI's GPT-5, alone with a basic prompt, scored 78.9 percent on the complex cases at about $8,400 per case. Wrapped in Microsoft's orchestration, it gained roughly four points — and got about $3,000 cheaper. With GPT-5.5, a newer model, DxO lifted accuracy from 72 to nearly 87 percent on the complex cases but added nothing on emergency-department cases, where both configurations scored about 85 at similar cost. The orchestration did its best work on weaker models — Google's Gemini 2.5 Flash jumped from 46 to nearly 69 percent — and the accuracy-point distance between a weak base model and a strong one was larger than the points Microsoft's panel added on top of any of them. Microsoft's own researchers concede the gains shrink as models approach the ceiling.

What survives that arithmetic is a procurement story. A Microsoft spokesperson told TechCrunch the orchestration helps with "optimization both accuracy and cost." Cheaper inference for mid-tier models is genuinely useful to a hospital's finance office. But the doctor-beating in these tables belongs to the base models, not the wrapper: GPT-5 does it alone, and what Microsoft added is a cost optimizer.

Dominic King, Microsoft's health vice president, offers the strongest defense. He told TechCrunch the biggest challenge in healthcare delivery is the growing mismatch between demand and supply of medical expertise. If that mismatch justifies the generalist baseline — measure the machine against the doctor a strained system can field — then the test should live where the strain lives: emergency departments. The Verge reports the system also beat human physicians there, 80 to 47 — a result the published paper achieved with GPT-5, on admissions data from Beth Israel Deaconess Medical Center that Microsoft compiled into SDBench. But per TechCrunch's reading of the tables, the orchestration added little on those emergency cases with GPT-5 and nothing with GPT-5.5, where both configurations scored about 85 percent at similar cost. Whatever doctor-beating happened in the emergency department, the base model did it unassisted. Suleyman himself concedes the distance left: "It needs clinical validation. That is a leap."

Microsoft already sells dictation and radiology tools to clinicians, and Suleyman tells TechCrunch the company must have "the absolute number one state-of-the-art medical clinical decision support products over the next few years." He is training a healthcare foundation model with the Mayo Clinic, foresees a 2027 breakthrough in streaming forecasts from medical records, expects an order-of-magnitude capability jump within 24 to 36 months, and describes the mechanism plainly: a flywheel, meaning a model in production, iterating on real-world usage. Real-world usage, in a hospital, is patient data. The gap between this study and this press release does work for one party: it sells the decision-support products Microsoft says it must lead, and it opens the hospital doors through which the flywheel's patient data flows.

Through a partnership with Beth Israel Deaconess Medical Center in Boston — the same hospital whose admissions data fed the benchmark — Microsoft will run DxO in parallel with standard care: consenting patients get normal treatment while the system makes its own recommendations, and the two are compared. King calls it potentially the biggest study of its kind. That trial, if it brings independent cases, specialists into the comparison, and patient outcomes rather than diagnostic elegance as the endpoint, is the number worth waiting for.

Until it arrives, read every percentage point from this week the way the study built its baseline: alone, without references, on cases chosen by the party selling the result. When a company announces it has beaten the physicians, the first audit question is which physicians were never in the room — and who decided.