Forwarded this by a colleague? Subscribe free and get it every week.
THIS WEEK
Three of this week's stories ask the same question from different angles: how much of a good result survives contact with a real clinic.
In this issue:
Microsoft's diagnostic AI scored 83% on NEJM cases against 36% for generalists. What the physicians were and weren't allowed to do.
FDA says draft generative-AI guidance is months away. The AMA and the AHA asked it for different things.
Spotlight: a randomized trial in which 20 hours of AI-literacy training did not stop physicians from following planted errors.
How-to: the Clinician's AI Tool Guide. Nineteen tools, rated on PHI, evidence and weak spots. Free.
LATEST NEWS
FDA's generative-AI draft guidance is months away. Hospitals and physicians asked for different things.
FDA's draft guidance on generative-AI devices is months away, Rick Abramson told the AdvaMed conference on Oct. 20, according to Elise Reuter at MedTech Dive. Abramson directs FDA's Digital Health Center of Excellence. Comments on the August discussion paper closed Oct. 19 with more than 500 submissions. Two concern you directly. The AMA's Oct. 16 letter, reported by Gus Iversen at DOTmed, backs risk-based oversight and wants physicians to help define what counts as a clinically significant failure, while saying physician review should not substitute for evidence or relieve manufacturers of accountability. The American Hospital Association's Oct. 19 letter also backs a risk-based framework and wants developers to carry post-market monitoring, but asks FDA to state that generative-AI clinical decision support and administrative tools fall outside its oversight.
Why it matters: The AHA wants generative-AI clinical decision support excluded from FDA oversight. For clinicians, the immediate question is local. Ask what evidence your hospital requires before it deploys a generative tool, and who monitors the ones already running.
Epic is exploring agents that could handle some routine visits
Sai Balasubramanian at Forbes, drawing on Becker's coverage of a Harvard Medical School informatics talk, reports that Epic executives described early work on AI agents that could gather a patient's initial information, order basic tests, and handle some routine visits under physician supervision. Garrett Adams, Epic's senior vice president for research and development, described a general-purpose engine for inpatient and outpatient settings. No product, timeline, or data were announced. Epic's stated motive is access: appointment delays, workforce shortages, and patients losing coverage.
Why it matters: Your EHR vendor is describing the move from documentation to action. Before an agent can order tests, establish who authorizes the order, who reviews the result and who handles follow-up. Ask your informatics lead whether your Epic contract already includes any agentic feature and who governs it.
Mayo uses an algorithm to decide which colonoscopies get an anesthesia provider
Patsy Newitt at Becker's ASC Review revisits remarks Mayo Clinic Health System president Prathibha Varkey made at a Becker's meeting in April. After a post-COVID CRNA shortage left a colonoscopy backlog, a gastroenterologist and one of Mayo's AI medical directors paired an evidence-based protocol with an algorithm that sorts patients between nurse-administered conscious sedation and anesthesia care. Varkey said Eau Claire, Wis., added about 150 colonoscopies a month within 30 days, and that the tool, in use for about five years, has spread across the system. No criteria, safety outcomes or publication were cited. The same piece quotes Henry Ford's anesthesiology chair calling GI sedation an expensive service to shed.
Why it matters: Expect your health system to use a triage algorithm to decide which endoscopy cases need you. Before it does, ask to see the sorting criteria, who can override them, and the sedation adverse-event rate before and after.
IN THE LITERATURE
Science — Nori et al.
Design: A Microsoft-built benchmark turns 304 NEJM clinicopathological conference cases and 325 MIMIC-IV emergency admissions into stepwise, text-only encounters: start from a short presentation, then request history, exam and tests one at a time, each test priced. MAI-DxO runs a language model (GPT-5 in the main analysis) as a simulated physician panel. Generalist physicians worked the same cases alone.
Finding: On 56 held-out NEJM cases, MAI-DxO reached 83.2% against 36.1% for generalists with reference tools and 22.3% without; the best single physician scored 51.8%. Emergency cases: 80.3% versus 46.8%. Across the baseline models tested, the orchestration layer added an average of 7.9 points on the NEJM cases and 4.4 on the emergency cases; most of the gap to physicians is the model itself, and the orchestration's clearer gain was cost.
Limitation: No patients, no images or waveforms, physicians working without colleagues or follow-up, NEJM cases chosen for teaching value, and correctness scored by another AI model. All authors are Microsoft employees with equity; eight are inventors on a related patent. The authors say the results do not establish superiority over specialists, teams or real practice.
Bottom line: The 83-versus-36 figure is Microsoft's number on Microsoft's benchmark, against physicians who could not phone a colleague. Treat it as performance on this benchmark. How it performs with real patients remains unknown. The result to wait for is the prospective study with real patients at Beth Israel Deaconess that Microsoft announced with the paper.
The BMJ — Chen et al.
Design: Systematic review of LLM clinical trials through Dec. 6, 2025: 68 trial reports, 15 protocols and 378 registrations, scored against TRIPOD-LLM, CONSORT 2025 and CHART.
Finding: Most reports were small, single-site 2024–25 trials of patient education or management advice, usually with GPT-4. Mean completeness was 40% on TRIPOD-LLM and 44% on CONSORT 2025. Of 68 reports, 47 (69%) were not preregistered, and only about a quarter of the 44 RCTs fully reported harms.
Limitation: Reporting quality is not trial quality; a well-run trial can be badly written up. The preregistration comparison is exploratory.
Bottom line: When a vendor cites a trial of its LLM tool, look for a registration number, the model version, and a harms section. Expect at least one to be missing.
Anesthesia & Analgesia — Hurley et al.
Design: Seven Stanford Emergency Manual crises (asystole, pulseless VT, unstable bradycardia, anaphylaxis, airway fire, malignant hyperthermia, LAST) rewritten as vignettes and put to 22 LLMs, which were asked to respond as the anesthesiologist. Three authors graded against rubrics that penalized harmful advice.
Finding: Models did best on general resuscitation (asystole, pulseless VT) and worst on the anesthesia-specific crises (LAST, airway fire). OpenAI's o1-preview scored highest and Llama 3.2 lowest. Medically fine-tuned models underperformed the general models they were built on.
Limitation: Vignettes, not simulation or clinical use. Published online March 2026, so the models are a generation old. Read through the APSF summary; the full text is paywalled.
Bottom line: The models were weakest on LAST and airway fire. The emergency manual stays on the wall.
SPOTLIGHT
Twenty hours of AI training did not stop physicians from following bad advice
Every AI governance plan I have seen leans on one safeguard: a trained clinician reviews the output. The AMA, in its letter above, says that review cannot replace evidence or manufacturer accountability. A randomized trial in NEJM AI examines what happens when trained physicians meet erroneous AI advice, and this month the journal ran a letter exchange on what to do about the result.
Qazi and colleagues at Lahore University of Management Sciences enrolled 44 physicians in Pakistan, all of whom had completed a 20-hour AI-literacy course. They randomized them 1:1 to work six diagnostic vignettes with ChatGPT-4o suggestions available on the side. The control group's suggestions were error-free. The treatment group's carried deliberately planted, clinically significant errors in three of the six cases. Consultation was voluntary; physicians kept their usual references and could accept, modify or ignore the AI. Both arms opened it in about two-thirds of cases. Three blinded physicians scored a composite of differential, supporting evidence, top diagnosis and next steps. The trial was preregistered (NCT06963957), which the BMJ review above found missing from most LLM trials.
Physicians who saw the erroneous suggestions scored 73.3% against 84.9% in the control group, an adjusted difference of −14.0 percentage points (95% CI −18.9 to −9.1). Top-choice accuracy fell from 90.5% to 76.1%, an adjusted −18.3 points (95% CI −26.6 to −10.0). These were people who had just been taught that this exact failure mode exists.
The design has limits the authors acknowledge. Vignettes are not patients. The errors were planted by physicians, not the ordinary kind a model produces on its own. One session, no arm without AI, 44 participants from one country. The subgroup signals (men and more experienced physicians drifted further) are small and descriptive. What the trial does show is that physicians who had just finished the course still followed planted errors. Because everyone was trained, it cannot say how much the course helped.
The October exchange is where it turns practical. Branda and Ciccozzi, statisticians at Campus Bio-Medico in Rome, argue that generic AI-literacy training is insufficient and that institutions should run empirical automation-bias stress tests, with governance that enforces vigilance. Qazi and Ali reply that training and governance are necessary but the missing piece is workflow design: when the AI output enters clinical reasoning matters as much as whether the clinician was trained to doubt it.
The takeaway: Errors still influenced physicians after training. One workflow worth testing: write your differential and next step before you open the AI, then ask it to challenge your reasoning. This trial did not test that sequence. If your institution's governance plan lists "clinician review" as the safeguard for a generative tool, ask whether anyone has tested it the way this trial did: feed the deployed tool a wrong answer and see who catches it. Until someone has, the safeguard is a hypothesis.
HOW-TO
Before you try this: check your organization's approved-tool list, and never put patient information into a tool it hasn't approved.
Nineteen AI tools, three questions each
Start with the tool you already use. The Clinician's AI Tool Guide, a free ten-page PDF, compares nineteen tools a practicing clinician can sign up for or ask for. Each is judged on patient information, published evidence for the product itself, and weak spots. Clinical reference: OpenEvidence, Doximity Ask, UpToDate Expert AI, ClinicalKey AI, Glass Health, Vera Health. Ambient scribes you can buy alone: Doximity Scribe, Freed, Heidi Health, Nabla. Scribes sold to health systems: Abridge, Ambience, Dragon Copilot, Suki. Literature search: Consensus, Elicit, Scite. General assistants with a clinician tier: ChatGPT for Clinicians, Claude. Prices and terms were read on the vendor pages on Oct. 23, 2026, and will drift.
Get the guide. Download it here. Free; claiming it adds you to this list if you are not already on it.
Find the tool you already use and read its PHI line first. Four labels: a BAA you can accept yourself, a BAA only through an institutional contract, terms that need review, and no BAA found. An individual BAA never by itself authorizes use with your employer's patients. That is the catch most clinicians miss with OpenEvidence, Doximity and ChatGPT for Clinicians.
Take the five compliance questions on page 2 to whoever owns your approved-tool list. Approved or not. Institutional BAA or not. Training, retention and the exact features covered. Where audio or text lives and for how long. Who gets a bad-output report.
Before you rely on any answer, run the five checks. The citation exists and says what the tool says. Doses and thresholds checked against a source you already trust. The answer fits this patient's meds, organ function and goals. Nothing invented, nothing important missing. Your judgment came first. For a scribe note, add medications, negations, laterality, who said what, and the exam you actually performed.
Tell me what is wrong. Corrections to [email protected] are credited in the next edition.
Nothing in the guide, and no tool in it, replaces your own check of a dose, a contraindication or a patient-specific decision. No vendor paid for placement or review, and I have no financial relationship with any company named. The guide was drafted with help from Claude, made by Anthropic; its entry is held to the same standard as the others.
THAT'S IT FOR THIS WEEK
How was this issue?
Send us a tool or a how-to. Using something in clinic that other clinicians should know about? Submit it here and it may show up in a future issue. We ask before we publish anything and credit you by name only if you want us to.
Know one colleague who’d use this? Forward this email, or send them the link below.
Loading Dose AI is for educational purposes only and is not medical advice. Nothing here should guide the care of a specific patient, and reading it does not create a physician-patient relationship. AI tools help gather and draft each issue; I read every source and edit every word. Opinions are mine and not my employer's. Full disclaimer