Microsoft's MAI-DxO in Science: an AI 'Panel of Doctors' That Out-Diagnosed Generalist Physicians 83% to 36%, and What It Still Can't Prove

What happened

On October 22, 2026, Microsoft AI published Building Medical Superintelligence, announcing that its diagnostic research is now peer-reviewed and published in Science as "Sequential diagnosis with language models". The paper introduces two things: SDBench, a benchmark that tests step-by-step diagnostic reasoning, and the Microsoft AI Diagnostic Orchestrator (MAI-DxO), the AI system that leads it. This page is the long-form briefing behind the BSH Technologies Instagram carousel.

• Headline result: 83% accuracy on complex NEJM cases, versus 36% for generalist physicians with textbooks and the internet, and 22% without them.
• Everyday cases: 80% versus 47% on common emergency-department presentations from MIMIC-IV-ED.
• Cost: a budget-conscious configuration kept about 81% accuracy while spending far less on tests.
• Next: a prospective study with real patients at Beth Israel Deaconess Medical Center in Boston.

1. Why a new benchmark was needed

Medical licensing exams such as the USMLE give all the facts up front and ask for one answer. Real diagnosis does not work that way: clinicians build the picture step by step, deciding what to ask, which tests to order and how each finding changes the likely diagnosis. Microsoft argues that exam-style tests have become the default yardstick for health AI even though they miss this process.

• SDBench turns cases into interactive puzzles. An AI model or a physician starts with limited information and must request more: questions, exams or tests.
• Two case sets: 304 complex cases from The New England Journal of Medicine's clinicopathological case records, and a set of more typical emergency-department admissions from the MIMIC-IV-ED database (325 cases, per Becker's Hospital Review).
• What it scores: diagnostic accuracy, the evidence requested, and the estimated cost of the tests ordered, so it judges both the answer and the path taken.

2. How MAI-DxO works: a virtual panel of physicians

• Instead of asking one model for an instant answer, MAI-DxO runs like a panel of virtual physicians. Each role proposes diagnoses, chooses useful questions or tests, or challenges the emerging view.
• The orchestrator coordinates several rounds of debate, updates its assessment as evidence arrives, and weighs each test's likely value against its cost.
• It is model-agnostic: the orchestration sits on top of language models from different labs. Microsoft says the gains persisted, and improved, when it re-ran the evaluation with newer models including GPT-5.5, without changing the method. Coverage of the paper reports the orchestration added about 7.9 points on complex cases and 4.4 points on ED cases across models.

3. The numbers

• Complex NEJM cases: MAI-DxO 83% (83.2%). Generalist physicians 36% (36.1%) with external references and 22% without, making MAI-DxO about 2.3x and 3.7x more accurate.
• Emergency-department cases: MAI-DxO 80% (80.3%) versus 47% (46.8%) for physicians, more than 1.7x.
• No one beat it on both axes: across both datasets, Microsoft says no other human or AI approach was both more accurate and less costly.

4. The cost side

• When tuned harder for cost, one configuration reached about 81% (80.9%) accuracy at roughly $2,241 per complex case, according to Becker's Hospital Review and Medical Economics, against about $4,725 for generalist physicians using reference tools. Microsoft describes it as less than half the cost of comparable systems.
• It asks before it tests. MAI-DxO asked an average of 25.5 questions before ordering its first test, compared with 5.6 for physicians, which is part of how it avoids expensive work-ups.

5. What comes next: real patients in Boston

• Beth Israel Deaconess Medical Center: in a new prospective study, patients will interact with the system directly and its performance will be measured against physicians' independent assessments. TechCrunch notes Google recently ran a similar real-world urgent-care study of its AMIE system with Beth Israel, published in The Lancet.
• Mayo Clinic: Microsoft is collaborating with Mayo Clinic to combine frontier AI, clinical expertise and longitudinal multimodal data; Mustafa Suleyman told TechCrunch the aim is a healthcare foundation model planned for next year.
• Open benchmark: Microsoft is open-sourcing SDBench for select NEJM cases, including the benchmark, the Gatekeeper and Judge agents, and cost-estimation code, so others can reproduce or challenge the results.
• One health team: Microsoft is folding health teams from Microsoft Research and the groups behind Dragon Copilot and PowerScribe into one effort at Microsoft AI. Its broader vision is set out in "The Building Blocks of Medical Superintelligence" in BMJ Digital Health and AI.

6. The catch: what this does not prove yet

• It is a benchmark. The results come from a controlled, text-only simulation. There is no physical exam, and no real patients have been diagnosed by MAI-DxO in a clinical study yet.
• The comparison is narrow. MAI-DxO was compared with individual generalist physicians, not specialists or care teams, on deliberately hard cases.
• Conflict of interest. All authors are Microsoft employees and Microsoft funded the work, so independent replication on the open-sourced SDBench cases will matter.
• Not a product. MAI-DxO is research, not a cleared clinical tool. Patients should keep relying on their own clinicians.

Why it matters for businesses

The lesson travels beyond medicine: orchestrating several AI roles that question, test and challenge each other, with an explicit cost budget, beat a single model answering in one shot. Teams building AI for support triage, fraud review or IT troubleshooting can borrow the same pattern: ask more before acting, price every expensive step, and measure the path, not just the final answer.

Sources

• Microsoft AI: Building Medical Superintelligence (Oct 22, 2026)
• Science: Sequential diagnosis with language models
• TechCrunch: Mustafa Suleyman says Microsoft's AI outperforms doctors on hard cases
• Becker's Hospital Review: Microsoft AI diagnostic tool tops generalist physicians, 83% to 36%
• Medical Economics: AI system outperforms physicians on diagnostic accuracy in Microsoft study
• Microsoft AI: The Path to Medical Superintelligence (2025 preprint announcement)
• arXiv: Sequential Diagnosis with Language Models (preprint)