Microsoft’s AI Doctor Claim: Separating The Headline From The Metric

🔍 Read the full analysis: Microsoft’s AI Doctor Claim: Separating The Headline From The Metric on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get privacy and security gear delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO diagnostic AI scored 83.2% on 304 difficult case records from the New England Journal of Medicine, compared with 36.1% for generalist physicians using textbooks and the internet, according to a Science paper published October 22. The peer-reviewed result is a benchmark finding, not evidence that the system is more accurate than doctors when treating real patients: the test used historical, text-only cases and compared the AI with individual generalists, not specialist teams.

The study also tested MAI-DxO on 325 emergency department cases, where it scored 80.3%, compared with 46.8% for physicians. On the difficult NEJM cases, doctors scored 22.3% without reference materials. These are results reported by Microsoft’s researchers for their own system; the Science publication makes the work peer-reviewed, but does not mean independent researchers have replicated the findings.

The benchmark, called the Sequential Diagnosis Benchmark, begins with a short case summary. A participant must request further information, such as a question or test result, from a simulated gatekeeper holding the case file. The benchmark evaluates the final diagnosis and the estimated cost of obtaining information. MAI-DxO uses a language model with a panel-style process in which virtual agents suggest diagnoses, choose questions or tests, and critique one another.

The headline system score does not represent the full gap between a general-purpose model and the doctors. Microsoft’s chart reports GPT-5 alone at 78.9% on the difficult cases, before the MAI-DxO orchestration layer was added. The source material says that layer added about four percentage points and lowered estimated test costs; GPT-5 alone was estimated to use $8,409 in tests per case. These figures are study estimates, not measured hospital spending.

What the Diagnostic Score Can Tell Us

The result suggests that language models may be able to organize information gathering and propose diagnoses effectively in a controlled setting. That could matter to hospitals and clinicians if future research shows similar performance with actual patients, where symptoms are incomplete, tests can be ambiguous, and decisions carry consequences.

For now, the score should not be read as proof that MAI-DxO can replace a doctor or outperform a clinical team. The benchmark’s participants were primary care doctors and hospital generalists working alone. In practice, difficult cases may be referred to specialists or discussed by teams. The study did not compare MAI-DxO with those approaches, so its results do not establish how it would perform against them.

The study also highlights a distinction between diagnostic accuracy and safe care. A high score on cases selected for diagnostic difficulty does not establish how often a system would raise false alarms, miss common benign explanations, or improve patient outcomes. Those questions matter to patients, insurers and health systems weighing whether to adopt or pay for clinical AI.

How Microsoft Built the Benchmark

The paper, Sequential diagnosis with language models, appeared in Science, volume 394, pages 423–428, on October 22, 2026. Its 15 authors are listed as Microsoft-affiliated, including Microsoft AI chief executive Mustafa Suleyman. An earlier version was posted on arXiv in June 2025 and used a different model and comparison; the source material says that preprint reported 80% for the AI, then using OpenAI’s o3, against 20% for doctors. The published paper’s reported figures differ, so the preprint numbers should not be substituted for the final study results.

Some methodological details in the supplied account come from that earlier preprint, and may not match the final paper. In the earlier version, a model judged diagnostic answers against the published diagnosis, with physician re-grading showing good but imperfect agreement. When a requested test result was absent from a case record, the gatekeeper generated a synthetic result. The preprint also estimated test prices using billing codes and a 2023 U.S. hospital price list, plus a fixed $300 physician visit charge; its authors described the costs as approximations.

The source account says the preprint’s physician participants included 21 U.S. and UK generalists, with a median of 12 years of experience. They were assessed on 56 held-out cases, while the AI was scored on all 304. Because these details refer to the preprint, they should be treated as background rather than assumed to describe every feature of the final Science study.

“1.7 times as accurate as physicians on emergency department cases, more than twice as accurate as physicians using textbooks and the internet on complex cases, and nearly four times as accurate as physicians without those resources.”

— Microsoft, in its blog post, “Building Medical Superintelligence”

What the Case Test Leaves Open

The study does not show how MAI-DxO performs in routine care with real patients, physical examinations or imaging. The benchmark used historical case records in a simulator. The source account also says the NEJM set contained difficult cases and no healthy patients or harmless explanations, limiting what can be concluded about false alarms. The system’s ability to distinguish rare disease from common, less serious causes remains untested by that case set.

It is also unclear whether the reported advantage would hold against specialist physicians, multidisciplinary teams, or clinicians using the same tools in a real workflow. The supplied account notes that the full Science paper is behind a paywall and that several detailed methods come from the earlier preprint, so some of those particulars may have changed. No independent replication is described in the source material.

Microsoft’s blog promotes the performance ratios, while the paper reports the study results. Neither the benchmark nor peer review alone establishes that using the system improves health outcomes, reduces costs in practice, or is safe for independent diagnosis. Those claims require evidence from clinical use.

The Planned Patient Study

The next development identified in the source material is a prospective study at Beth Israel Deaconess Medical Center in Boston. Consenting patients are expected to interact with the system alongside their usual care. That design could provide evidence about how the tool behaves in a clinical setting, although the source account does not specify the study’s start date, enrollment target, measures or results timeline.

Readers should look for details on whether the research evaluates diagnostic accuracy, safety, clinician decision-making, patient outcomes and costs, and whether results are assessed by independent investigators. Until those findings are available, MAI-DxO’s 83.2% score is best understood as performance on a selected benchmark—not a measured success rate for diagnosing patients in hospitals.

Key Questions

What did Microsoft’s MAI-DxO score 83.2% on?

It scored 83.2% on 304 difficult NEJM case records in the benchmark reported in Science. The cases were historical and presented through a text-based simulation, rather than encounters with actual patients.

Did the study show that the AI is better than doctors?

It found higher scores than individual generalist physicians in the tested comparisons. It did not compare MAI-DxO with specialists or physician teams, and it did not establish that the system performs better in real-world patient care.

Was the research peer-reviewed?

Yes. The paper, Sequential diagnosis with language models, was published in Science on October 22, 2026. Peer review does not by itself mean the results have been independently reproduced.

Did MAI-DxO account for the entire AI score?

No. Microsoft’s reported chart gives GPT-5 alone a 78.9% score on the difficult cases. The source material says MAI-DxO’s orchestration layer added about four percentage points and reduced estimated test costs.

What happens next?

The source material identifies a planned prospective study at Beth Israel Deaconess Medical Center, where consenting patients would use the system alongside their usual care. Its timeline and detailed evaluation measures have not been provided.

Source: ThorstenMeyerAI.com