“From Triage To Discharge”: Do New LLM Versions Handle the ED Better? Skyer Answers.
Shadi Nourbakhsh1, Mehrdad Majzoobi1
1Skye Insights Ltd., Toronto, ON, Canada
LLM/VLM Deployment Opportunities and Risks in Healthcare (DAIH), a workshop at the Conference on Language Modeling (COLM) 2026, San Francisco, October 9, 2026. · Peer-reviewed, non-archival · OpenReview
This paper was peer-reviewed for the workshop and accepted on a non-archival track: it is not published in formal proceedings and carries no DOI. It builds on the Skyer benchmark reported in Canadian Journal of Emergency Medicine.
Abstract
Generative AI models undergo frequent updates, but for clinical implementers, each release triggers a significant and costly dilemma that standard benchmarks fail to address: does the new version truly enhance performance for their specific workflows, and does it justify the associated costs and risks? We previously introduced Skyer, a 55-scenario, clinician-validated benchmark for pediatric Emergency Department (ED) triage. Here we generalize it into a reusable framework for safety-aware, version-over-version evaluation of LLMs across diagnosis, triage, and decision-making performance, plus their consistency. In Skyer, rather than rewarding raw accuracy, we applied a weighting system to reward correct scores while penalizing both over- and under-scoring errors to capture the real-world patient-safety impact of mistriage, paired with a consistency criterion for reliability. Applying Skyer to fifteen models (eight GPT, five Gemini, and two Gemma versions) as an empirical demonstration, we find that newer is frequently not better: Gemini improves on a gradual steady path, Gemma remains on a low and inconsistent plateau, and GPT improves erratically, with an older version outperforming its successor. These findings show why upgrade decisions need a per-release instrument rather than vendor benchmarks, and we distil them into a deployment protocol that weighs each release's safety-weighted gain against its incremental cost. Because Skyer is contamination-resistant and version-agnostic, it extends to future releases and, by re-parameterization, to other clinical tasks.
Keywords Large language models · Emergency department tasks · Model iteration and increments · Version-over-version framework · Version upgrade decision · Cost-benefit · Patient safety · Model deployment
What this paper adds
A reusable framework.
Safety-aware, version-over-version upgrade decisions that transfer to future releases and to other clinical tasks.
An extension of Skyer's safety-weighted scoring.
From triage alone to diagnosis and decision-making, so the same patient-safety cost accounting applies across all three ED tasks.
An empirical evaluation.
Across fifteen models (eight GPT, five Gemini, and two Gemma versions), showing that newer is frequently not better. In this evaluation the older GPT-4.5-preview outperformed the newer GPT-5, and Gemini-1.5-pro outperformed the updated Gemini-2.5_03-25.
Deployment protocol
Skyer turns model-version selection into a repeatable check a deployer runs on each release:
1. Run Skyer on the candidate version against the incumbent for the target task.
2. Require a safety-weighted gain over the incumbent, not merely higher accuracy.
3. Require consistency of at least 80%.
4. Weigh any gain against the incremental cost of the new version.
5. Emit a verdict: upgrade, hold, freeze, or roll back.
Data availability
Two datasets are released on Hugging Face under CC BY-NC-ND 4.0. The Data page has the full description.
Skyer benchmark results are openly available: per-case triage predictions, disposition decisions, and diagnostic-accuracy aggregates for 16 model versions across the 55 scenarios, alongside the reference standard. huggingface.co/datasets/SkyeInsights/skyer-results
A five-scenario sample of the prompt set is access-gated and granted by approval, for evaluation only. huggingface.co/datasets/SkyeInsights/skyer-sample-prompts
The complete 55-scenario set is withheld to keep the benchmark contamination-resistant. Decoding parameters, query dates, and per-run outputs are not yet included.
How to cite
Nourbakhsh, S., Majzoobi, M. “From Triage To Discharge”: Do New LLM Versions Handle the ED Better? Skyer Answers. LLM/VLM Deployment Opportunities and Risks in Healthcare (DAIH), workshop at COLM 2026. https://openreview.net/forum?id=okLCTCAB7C
@inproceedings{nourbakhsh2026from,
title = {``From Triage To Discharge'': Do New {LLM} Versions
Handle the {ED} Better? Skyer Answers.},
author = {Shadi Nourbakhsh and mehrdad majzoobi},
booktitle = {LLM/VLM Deployment Opportunities and Risks in Healthcare},
year = {2026},
url = {https://openreview.net/forum?id=okLCTCAB7C}
}