Abstract bekijken

Deze abstract is toegekend aan sessies
Type presentatieOral presentation
TitelGuideline Adherence of General-Purpose and Retrieval-Augmented AI Chatbots in Glaucoma Care: A Benchmark Study Against the EGS Guidelines
DoelTo assess the concordance of eight publicly available artificial intelligence (AI) chatbots with four current European Glaucoma Society (EGS) reference works, and whether direct access to these documents improves performance.
MethodesIn this cross-sectional benchmark study, 239 expert-validated question–answer sets derived from four EGS reference works were submitted to six general-purpose (ChatGPT, Claude, Copilot, DeepSeek, Gemini, Grok) and two medically oriented retrieval-augmented (OpenEvidence, Vera Health) chatbots under three knowledge conditions: naive, guideline instruction without document provision, and provision of the full-text EGS reference work. Two masked investigators graded responses as accurate or inaccurate; the primary endpoint was accuracy in the naive condition.
ResultatenOverall accuracy was 77·0% (4047/5258). Naive accuracy was 78·1% (1493/1912), ranging from 71·5% (Gemini) to 82·4% (OpenEvidence) (P<0·001). Provision of the full-text reference work raised accuracy to 86·2% (1236/1434; adjusted OR 3·05, 95% CI 2·41–3·85; P<0·001), reaching 95·8% for Claude, whereas guideline instruction alone did not improve accuracy. Accuracy was lowest for open-ended questions (68·3%) and for questions from the EGS Guide on Surgical Innovation for Glaucoma (60·9%).
ConclusieWithout direct access to the relevant EGS material, chatbots answered approximately one in five guideline-derived glaucoma questions incorrectly. Direct provision of authoritative guideline material substantially improved guideline-concordant performance, although the benefit varied between models. Grounding is therefore essential.
BelangenverstrengelingNee
Details verstrengelde belangenNo relevant COIs
Auteurs 1
NaamJACOBS
InitialenT
InstituutUZ/KU Leuven
StadLeuven
Auteurs 2
NaamPonette
InitialenL
InstituutKU Leuven
StadLeuven
Auteurs 3
NaamColson
InitialenA
InstituutKU Leuven
StadLeuven
Auteurs 4
NaamVan Eijgen
InitialenJ
InstituutUZ/KU Leuven
StadLeuven
Auteurs 5
NaamStalmans
InitialenI
InstituutUZ/KU Leuven
StadLeuven
top ^