THE BELGIAN OPHTHALMIC ASSOCIATIONS
View abstract
Cet abstract a été assigné à sessions
| Type de présentation | Oral presentation |
| Titre | Guideline Adherence of General-Purpose and Retrieval-Augmented AI Chatbots in Glaucoma Care: A Benchmark Study Against the EGS Guidelines |
| But | To assess the concordance of eight publicly available artificial intelligence (AI) chatbots with four current European Glaucoma Society (EGS) reference works, and whether direct access to these documents improves performance. |
| Méthodes | In this cross-sectional benchmark study, 239 expert-validated question–answer sets derived from four EGS reference works were submitted to six general-purpose (ChatGPT, Claude, Copilot, DeepSeek, Gemini, Grok) and two medically oriented retrieval-augmented (OpenEvidence, Vera Health) chatbots under three knowledge conditions: naive, guideline instruction without document provision, and provision of the full-text EGS reference work. Two masked investigators graded responses as accurate or inaccurate; the primary endpoint was accuracy in the naive condition. |
| Résultats | Overall accuracy was 77·0% (4047/5258). Naive accuracy was 78·1% (1493/1912), ranging from 71·5% (Gemini) to 82·4% (OpenEvidence) (P<0·001). Provision of the full-text reference work raised accuracy to 86·2% (1236/1434; adjusted OR 3·05, 95% CI 2·41–3·85; P<0·001), reaching 95·8% for Claude, whereas guideline instruction alone did not improve accuracy. Accuracy was lowest for open-ended questions (68·3%) and for questions from the EGS Guide on Surgical Innovation for Glaucoma (60·9%). |
| Conclusion | Without direct access to the relevant EGS material, chatbots answered approximately one in five guideline-derived glaucoma questions incorrectly. Direct provision of authoritative guideline material substantially improved guideline-concordant performance, although the benefit varied between models. Grounding is therefore essential. |
| Conflit d'intérêt | Non |
| Détails conflits d'intérêt | No relevant COIs |
Auteurs 1
| Nom | JACOBS |
| Initiales | T |
| Institut | UZ/KU Leuven |
| Ville | Leuven |
Auteurs 2
| Nom | Ponette |
| Initiales | L |
| Institut | KU Leuven |
| Ville | Leuven |
Auteurs 3
| Nom | Colson |
| Initiales | A |
| Institut | KU Leuven |
| Ville | Leuven |
Auteurs 4
| Nom | Van Eijgen |
| Initiales | J |
| Institut | UZ/KU Leuven |
| Ville | Leuven |
Auteurs 5
| Nom | Stalmans |
| Initiales | I |
| Institut | UZ/KU Leuven |
| Ville | Leuven |