View abstract

This abstract is assigned to sessions

Presentation type	E-poster
Title	Inter-observer variability of human optic disc evaluation on fundus photos and implications for AI-research

Purpose	To evaluate the inter-observer variability in grading of optic discs from colour fundus photos (with or without intra-ocular pressure, IOP), in order to explore the robustness of ground truth labels based solely on optic disc scoring for external validation of AI-algorithms for glaucoma detection.
Methods	Fundus photos were obtained from a cohort of 69 individuals. The parameters were age, bilateral fundus pictures and IOP. Colour images of 138 optic nerves were analyzed separately by 5 ophthalmologists (4 glaucoma specialists and 1 ophthalmology resident) and Vertical Cup-to-Disc Ratio (VCDR) was recorded for each optic nerve. Also, the eyes were classified into 3 categories (glaucomatous, glaucoma suspect or healthy), initially without knowing the IOP and subsequently again after being unblinded for IOP. Intraclass correlation coefficients (ICC) were used to evaluate agreement in VCDR scoring on the image level. Quadratic Weighted Fleiss score was used to evaluate agreement on the categorization into the 3 categories on the patient level.
Results	For the inter-rater agreement in VCDR scoring, the ICC was 0.62 (95% CI, 0.51-0.72) (moderate). For the categorization into 3 classes (healthy, suspect or glaucoma), the Quadratic Weighted Fleiss score was 0.568 (95% CI, 0.421-0.714) (moderate) when blinded for the IOP and 0.441 (95% CI, 0.292-0.59) (moderate) after revealing the IOP.
Conclusion	This study indicates a high inter-observer variability with moderate agreement based solely on color fundus pictures, even if evaluated by glaucoma specialists. This represents a limitation to the validation of AI algorithms when the ground truth has only been determined on fundus photos.
Conflict of interest	No

Authors 1

Last name	BOURGUIGNON
Initials	G
Department	Cliniques universitaires Saint-Luc
City	Bruxelles

Authors 2

Last name	POURJAVAN
Initials	S
Department	Cliniques universitaires Saint-Luc
City	Bruxelles