Keikku
    Back to Blog

    AI Aces the USMLE, Then Falls Apart at the Bedside: Why Every Clinical AI Needs a Physician in the Loop

    Posted in Keikku Blog

    By Diana van Stijn, MD PhD

    June 5, 2026


    AI Aces the USMLE, Then Falls Apart at the Bedside: Why Every Clinical AI Needs a Physician in the Loop

    Twelve of the most capable Artificial Intelligence (AI) models in the world were handed a set of medical board questions. They scored brilliantly, as they always do. Then researchers made one small change: they removed the correct answer from the list of options, or added a choice that said "none of the above." The models did not pause. They did not say "I am not sure." They confidently picked a wrong answer anyway, over and over, as if nothing had changed.

    By Diana van Stijn, MD PhD | Augmented by Keikku AI

    That finding, published in Nature Communications in 2025, should be printed and taped to the monitor of every hospital executive currently being sold a fully autonomous AI. It exposes the single most dangerous illusion in medicine right now: that scoring well on a test means you can be trusted with a patient.

    The test these models pass is not the job they would be doing

    When a vendor tells you their model performs at the level of a top student on the United States Medical Licensing Examination (USMLE), what they are describing is success on an artificially constructed scenario. A board question is a sealed world. It hands over a clean vignette, promises that exactly one of the listed options is correct, and asks the model to pattern-match its way there. Large language models are superb at this, because the format rewards recognition rather than genuine reasoning. The answer is always already in the room, waiting to be matched.

    The researchers behind the Nature Communications study built a benchmark called MetaMedQA precisely to break that illusion. They took standard questions and altered the structure, removing the right answer or introducing an "I cannot answer" or "none of the above" option. Across all twelve models tested, the systems consistently failed to recognize the limits of their own knowledge and produced confident answers even when no correct option existed. The authors named the problem directly: a critical disconnect between what these models appear to know and what they actually know, posing significant risks in clinical settings.

    A separate research team made the point even more bluntly. They took 100 standard board questions and simply swapped the correct answer for "none of the other answers," then had a clinician confirm that this was now the right response. Accuracy collapsed. Depending on the model, the share of these modified questions answered incorrectly ran from roughly 9% to nearly 38%. The medicine had not changed. The only thing that changed was that the answer was no longer sitting in the lineup to be recognized, and that was enough to break the model.

    Real patients are nothing but altered variables

    Here is why this matters far beyond a benchmarking exercise. A board exam is the most controlled environment a clinical problem will ever appear in. Real patients are the opposite. They present with the atypical symptom, the confounding comorbidity, the medication nobody documented, the detail that does not fit the pattern. Real medicine is the MetaMedQA test running continuously, with the variable changed every single time.

    This lines up with what the broader evidence shows. A State of Clinical AI report from Stanford and Harvard found that AI systems which match or beat physicians on fixed, well-framed cases see their performance fall when they must ask follow-up questions, manage incomplete information, or revise a decision as new details emerge. Under genuine uncertainty, AI has been observed to reason more like a medical student than an experienced physician, committing hard to an answer when the right move is doubt. A 2026 review in the journal Diagnostics reached the same destination, concluding that AI will augment rather than replace physicians, citing models' tendency toward hallucination and overconfidence, their inability to perform a physical examination, and the unresolved question of who is liable when an autonomous system is wrong.

    The throughline across all of these is the same trait MetaMedQA isolated: current AI is most dangerous precisely when it is most confident and most wrong. And confidence without calibration is not a quirk you can patch around. It is the core reason the technology cannot be left alone with a patient.

    A physician in the loop is not a courtesy. It is the safety mechanism.

    This is what "physician in the loop" actually means, and why it is non-negotiable. The physician is not there to rubber-stamp the AI's output or to slow things down. The physician is the component that supplies exactly what the models lack: the ability to notice when the case has left the textbook. A clinician is trained to feel the wrongness of a presentation that does not add up, to hold uncertainty instead of forcing a confident answer, to recognize the variable that was not supposed to be there. That is the precise capability MetaMedQA proved the models do not have.

    Take the physician out of the loop and you do not get faster medicine. You get a system that performs beautifully right up until the moment a real patient introduces a variable it was not expecting, at which point it fails silently and confidently, with no internal signal that anything is wrong. In a board exam, a confident wrong answer costs a point. At the bedside, it costs a patient.

    How to build AI that is actually safe to use

    None of this is an argument against clinical AI. It is an argument about how to build and deploy it. The right model is augmentation: let AI carry the load it genuinely excels at while keeping the physician as the reasoning and accountability layer.

    That means an AI Scribe that drafts the documentation and lets the clinician confirm it, rather than filing notes unsupervised. It means Reference and Clinical Decision Support (CDS) that surface referenced, current answers for a physician to weigh, rather than dictating a course of action. It means Diagnostics AI, including AI Auscultation that analyzes heart and lung sounds, that flags a possible finding for the clinician to verify, rather than rendering a verdict. And it means Voice AI that keeps the doctor present with the patient instead of buried in a screen. In every one of these, the AI proposes and the physician disposes. The intelligence is real and useful. The judgment stays human.

    The headline you should actually worry about

    The dangerous headline is not "AI passes the medical licensing exam." The dangerous headline is the one that quietly follows it: a health system deploying that exam-passing model as if the score were a safety certificate. MetaMedQA is the proof that it is not. These systems are extraordinary pattern-matchers operating inside controlled frames, and patients do not live inside controlled frames.

    AI's inaccuracy in real clinical situations is not a hypothetical or a fear. It is a measured, replicated finding. The responsible conclusion is not to reject the technology but to refuse to remove the one safeguard that catches it when it fails. Keep the physician in the loop. The AI is the instrument. The physician is the one who knows when the instrument is lying.

    This is exactly how Keikku Reference is built

    Keikku Reference is a clinical reasoning AI built around the lesson MetaMedQA teaches. It is not designed to render verdicts or to act on a patient unsupervised. It is designed to augment the physician, surfacing referenced, current clinical answers so the clinician can reason faster and with better evidence, while the judgment and the accountability stay exactly where they belong.

    Three things make that augmentation safe rather than a liability. First, Reference shows its work. Every answer is grounded in cited sources, so the physician can see the basis for a claim and check it, rather than trusting a confident sentence with no provenance, which is precisely the failure mode the research exposes. Second, Reference is context-aware when it matters. When embedded in the clinical encounter, it reasons over the actual patient, the documentation captured by the AI Scribe and the findings flagged by Diagnostics AI, instead of answering a generic textbook question in the abstract. And third, it keeps the physician as the decision-maker by design. Reference proposes; the clinician disposes.

    That is the whole philosophy in one product. The intelligence is real, current, and genuinely useful at the point of care. The physician remains the reasoning layer that catches the variable the model was not expecting. Keikku does not ask you to choose between the speed of AI and the safety of human judgment. It is built so you do not have to.


    Sources

    1. Griot M, et al. "Large Language Models lack essential metacognition for reliable medical reasoning." Nature Communications, 2025. https://www.nature.com/articles/s41467-024-55628-6
    2. "Fidelity of Medical Reasoning in Large Language Models" (None of the Other Answers substitution study). PMC, 2025. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12334947/
    3. Stanford Medicine. "Clinical AI Has Boomed. A New Stanford-Harvard State of Clinical AI Report Shows What Holds Up in Practice." https://medicine.stanford.edu/news/current-news/standard-news/clinical-ai-has-boomed.html
    4. Obuchowicz R, Piórkowski A, Nurzyńska K, Obuchowicz B, Strzelecki M, Bielecka M. "Will AI Replace Physicians in the Near Future? AI Adoption Barriers in Medicine." Diagnostics, 2026. https://pmc.ncbi.nlm.nih.gov/articles/PMC12897247
    5. American Medical Association. "Using AI to heal the system, not replace the physician." https://www.ama-assn.org/practice-management/digital-health/using-ai-heal-system-not-replace-physician