.Nature Medicine, Released online: 26 July 2024 doi:10.1038/ s41591-024-03181-6Although sizable language models (LLMs) reveal promise in regulated settings, a study now reveals their limits in real-world professional uses and also directs the method towards robust analysis and benchmarking before clinical usage.