View abstract

This abstract is assigned to sessions

Presentation type	Oral presentation
Title	Guideline Adherence of General-Purpose and Retrieval-Augmented AI Chatbots in Glaucoma Care: A Benchmark Study Against the EGS Guidelines

Purpose	To assess the concordance of eight publicly available artificial intelligence (AI) chatbots with four current European Glaucoma Society (EGS) reference works, and whether direct access to these documents improves performance.
Methods	In this cross-sectional benchmark study, 239 expert-validated question–answer sets derived from four EGS reference works were submitted to six general-purpose (ChatGPT, Claude, Copilot, DeepSeek, Gemini, Grok) and two medically oriented retrieval-augmented (OpenEvidence, Vera Health) chatbots under three knowledge conditions: naive, guideline instruction without document provision, and provision of the full-text EGS reference work. Two masked investigators graded responses as accurate or inaccurate; the primary endpoint was accuracy in the naive condition.
Results	Overall accuracy was 77·0% (4047/5258). Naive accuracy was 78·1% (1493/1912), ranging from 71·5% (Gemini) to 82·4% (OpenEvidence) (P<0·001). Provision of the full-text reference work raised accuracy to 86·2% (1236/1434; adjusted OR 3·05, 95% CI 2·41–3·85; P<0·001), reaching 95·8% for Claude, whereas guideline instruction alone did not improve accuracy. Accuracy was lowest for open-ended questions (68·3%) and for questions from the EGS Guide on Surgical Innovation for Glaucoma (60·9%).
Conclusion	Without direct access to the relevant EGS material, chatbots answered approximately one in five guideline-derived glaucoma questions incorrectly. Direct provision of authoritative guideline material substantially improved guideline-concordant performance, although the benefit varied between models. Grounding is therefore essential.
Conflict of interest	No
Details of conflicting interests	No relevant COIs

Authors 1

Last name	JACOBS
Initials	T
Department	UZ/KU Leuven
City	Leuven

Authors 2

Last name	Ponette
Initials	L
Department	KU Leuven
City	Leuven

Authors 3

Last name	Colson
Initials	A
Department	KU Leuven
City	Leuven

Authors 4

Last name	Van Eijgen
Initials	J
Department	UZ/KU Leuven
City	Leuven

Authors 5

Last name	Stalmans
Initials	I
Department	UZ/KU Leuven
City	Leuven