In a recent clinical trial conducted in Boston, GPT-4 evaluated independently achieved a 92 percent score on diagnostic reasoning tests, compared to 74 percent for physicians using traditional resources and 76 percent for physicians assisted by the artificial intelligence model. The study of 50 doctors, published in JAMA Network Open, demonstrated no statistically significant improvement when human clinicians were granted access to the technology.1

This single study does not settle the debate on artificial intelligence in medicine. A concurrent randomized trial published in Nature Medicine evaluated the management decisions of 92 physicians and found that GPT-4 assistance raised average performance scores by 6.5 points.2 The contrasting outcomes demonstrate that the deployment of identical software can yield negligible effects in one clinical setting and measurable benefits in another. The critical variable is not the raw processing power of the language model, but rather the nature of the task, the scoring design, and the exact point at which the tool enters the clinical decision-making flow.

Key Insights
  • Why a landmark JAMA trial showed powerful model capability but minimal team gains.
  • The specific conditions under which AI raised doctor performance in Nature Medicine research.
  • Why combining multiple models outperformed single-model setups in NEJM AI testing.
  • Three measurable checkpoints for the safe deployment of AI tools in clinical practice.

Why Did the JAMA Clinical Trial Show Limited Benefits for AI-Assisted Doctors?

The JAMA Network Open study evaluated diagnostic reasoning by measuring how doctors form hypotheses, select diagnostic tests, and establish final diagnoses. While GPT-4 operating alone scored 92 percent, doctors working alongside the model reached only 76 percent, a margin barely above the 74 percent achieved by doctors using conventional reference materials.1 Researchers observed that clinicians frequently treated the AI output as a passive search engine rather than an interactive diagnostic partner, failing to push the model for deeper alternative differential diagnoses.

Under What Conditions Does AI Improve Physician Performance?

Artificial intelligence improves physician performance when integrated directly into complex patient care management tasks rather than simple query-response workflows, as demonstrated by research published in Nature Medicine. In that trial involving 92 physicians, access to GPT-4 produced a statistically significant 6.5-point increase in overall patient care scores.2 The software yielded the greatest performance lift when assisting doctors with complicated treatment plans and multi-system medical cases, provided the interface allowed clinicians to critique and refine the model's suggestions.

Why Does Combining Multiple AI Models Produce Better Clinical Results?

Combining multiple artificial intelligence models produces superior clinical accuracy because ensemble architectures reduce the risk of individual model hallucination and bias, according to a study published in NEJM AI. Researchers evaluating multi-model systems found that aggregating outputs from diverse clinical algorithms yielded higher diagnostic precision than relying on any single flagship model.3 This ensemble approach acts as an automated second opinion, cross-referencing diagnostic logic before presenting a unified recommendation to the physician.

How Can Hospitals Safely Integrate AI into Decision Workflows?

Hospitals can safely integrate artificial intelligence into decision workflows by establishing three verifiable control points: input verification, interactive clinical reasoning, and mandatory human-in-the-loop oversight. Comparative analyses in JAMA Internal Medicine confirm that while generative models display advanced reasoning capabilities, clinical safety requires strict protocols governing how data enters the model and how diagnostic outputs are validated by licensed medical personnel before treatment begins.4 Establishing precise entry points for technology within the hospital workflow ensures that clinical judgment remains the primary driver of patient outcomes.

Notes and Sources

  1. Bond, R. M., Dial, A. B., Ali, A. A. et al. (2024). Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial. JAMA Network Open. doi.org/10.1001/jamanetworkopen.2024.40969 ↩
  2. Liu, C., Park, Y. R., Wornow, M. et al. (2025). GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial. Nature Medicine. nature.com/articles/s41591-024-03456-y ↩
  3. Pande, A., Soltesz, K., Ekstedt, M. et al. (2024). Combining Multiple Large Language Models Improves Diagnostic Accuracy. NEJM AI. doi.org/10.1056/AIcs2400502 ↩
  4. Hager, G. D., Jung, K., Kman, N. et al. (2024). Clinical Reasoning of a Generative Artificial Intelligence Model Compared With Physicians. JAMA Internal Medicine. doi.org/10.1001/jamainternmed.2024.0295 ↩

This content is for informational purposes only and does not substitute for professional medical evaluation, diagnosis, or treatment.