Skyer: a novel benchmark for evaluating the effectiveness of large language models in emergency department triage
Shadi Nourbakhsh1
1Skye Insights Ltd., Toronto, ON, Canada
Canadian Journal of Emergency Medicine (2026) · doi.org/10.1007/s43678-026-01214-2 · Open Access · CC BY-NC-ND 4.0
Abstract
Objectives Emergency department (ED) overcrowding causes diagnostic challenges, prolonged wait times, and impairs appropriate triage, often due to human error and fatigue. Large language models can assist ED staff in triage, improving patient care by mitigating these problems.
Methods We designed an evaluation method (Skyer benchmark) to assess fifteen large language models, including DeepSeek-R1 (70B, 7B), ChatGPT versions (4, 4.5-preview), Gemini iterations (1.5-pro, 2.0-Pro-experimental, 2.5-03-25, 2.5-05-06), Mistral-7B, Llama-3.3, Gemma models (2-27b-it, 3-12b-it, 3-27b-it), Qwen-2.5, and Phi-4-14B. We assessed the performance of models in ED triage by using 55 realistic clinical pediatric scenarios. The main objective was to evaluate models' triage performance using a weighting system that accounted for the impacts of over-triage and under-triage, in addition to simple accuracy. A secondary objective was to assess models' consistency, by repeating tests across scenarios three times.
Results Our findings indicate that only two models, ChatGPT-4.5-preview and Gemini-2.5-05_06, demonstrated superior and reliable triage performance. ChatGPT-4.5-preview (77% accuracy, mean weight 377.5 out of 550) and Gemini-2.5-05_06 (74% accuracy, mean weight 365/550) significantly outperformed human-triage-experts accuracy (64% accuracy, mean weight 253.5/550) and other models. This difference was statistically significant (p-value <0.05), with an extremely large effect size (Cohen's D=2.18 and 1.98). Furthermore, they demonstrated sufficient reliability due to acceptable consistency in their triage performance (85% and 82%).
Conclusion Our work establishes Skyer as an evaluation approach of large language models. Skyer selected the best-performing models, which demonstrated significantly higher triage performance than the human experts and showed consistent results. These large language models can streamline the delivery of quality healthcare in overcrowded EDs. Despite promising outcomes, we identified limitations prohibiting these large language models as replacement of human experts. Instead, we demonstrate their potential for a substantial role in assisting the staff in overcrowded EDs.
Keywords Artificial intelligence (AI) · Large language models · Pediatrics emergency department (ED) · Triage · Benchmark
Clinician's capsule
What is known about the topic?
Overcrowded EDs, particularly pediatric EDs, present significant challenges to healthcare systems, resulting in prolonged waiting times, misdiagnoses and improper triage.
What did this study ask?
We designed the Skyer benchmark to assess large language models performance as a reliable triage assistant in real-life ED situations.
What did this study find?
This study reveals that two of fifteen evaluated models can assist triage-experts performance, reduce errors and improve ED flow.
Why does this study matter to clinicians?
Skyer can evaluate current and upcoming models to determine the most effective assistance for ED performance leading to improved patient care.
How to cite
Nourbakhsh, S. Skyer: a novel benchmark for evaluating the effectiveness of large language models in emergency department triage. Can J Emerg Med (2026). https://doi.org/10.1007/s43678-026-01214-2
@article{Nourbakhsh2026Skyer,
author = {Nourbakhsh, Shadi},
title = {Skyer: a novel benchmark for evaluating the effectiveness of
large language models in emergency department triage},
journal = {Canadian Journal of Emergency Medicine},
year = {2026},
doi = {10.1007/s43678-026-01214-2},
url = {https://doi.org/10.1007/s43678-026-01214-2}
}