Artificial Intelligence in Retina
Artificial Intelligence (AI) refers to software that can mimic cognitive functions such as learning and problem solving by processing and recognizing patterns in large amounts of data. Machine learning, a subset of AI, is a method commonly utilized in the development of automated diagnostic systems. In machine learning, the machine creates its own algorithms by “learning” the associations between the input and the output. A range of analytical techniques may be incorporated in machine learning. The use of artificial neural networks is one common example. In essence, the machine is presented with an input (e.g. an image with features previously identified through traditional methods), and the machine processes this information and produces the output (e.g. the diagnosis).
Deep learning is a subset of machine learning, where the simple artificial neural network is expanded to include many hidden layers between the input and the output layers, and this is called the Convolutional Neural Network(CNN), which allows for more intricate evaluation of the input. Deep learning can be further classified into supervised, semi-supervised, and unsupervised learning. This allows the deep learning system to detect (e.g. disease) features on its own that may not be visible to the human eye or known by experts. Machine learning algorithms, and deep learning systems in particular, have been developed to improve diagnosis and management of various disease entities in ophthalmology.
Imaging Modalities and Image Processing Techniques in Retina
Retina specialists can utilize numerous imaging techniques for the diagnosis and treatment of retinal diseases, including optical coherence tomography (OCT), OCT Angiography, fundus photography, fundus autofluorescence, and others. Each patient comes with a wealth of diagnostic information that can be applied to deep learning analysis. The most common application of AI methods in retina is the detection of disease-related features on color fundus photographs. With the advent of OCT in the early 2000s, automated segmentation has been applied to delineate anatomic and pathologic features in retinal diseases. The use of a threshold can be used to segment retinal vessels, which are typically darker than their surroundings. By setting a threshold above a certain pixel intensity, certain pixels can be accepted as belonging to a vessel.
The application of deep learning to automated segmentation has enabled accurate detection of features such as intraretinal fluid, subretinal fluid, drusen, pigment epithelial detachment, and geographic atrophy with comparable accuracy as human graders. OCT Angiography (OCTA) utilizes sequentially acquired OCT-B scans to construct a depth-resolved image of retinal vasculature. Alam et al analyzed quantitative vascular features on OCTA images using a supervised machine learning algorithm and demonstrated that it was able to differentiate stages of diabetic retinopathy, sickle cell retinopathy, and disease versus. control.
Applications of AI in Retina
Pattern recognition in retinal imaging modalities is well-suited for the application of artificial intelligence to image analysis. It has been implemented in the screening and diagnosis of retinal diseases such as Diabetic Retinopathy (DR), Retinopathy of Prematurity (RoP) and Age-related Macular Degeneration (AMD).
Validated AI systems for the assessment of DR have been reported to perform diagnostic functions with high sensitivity and specificity. In particular, AI algorithms have been shown to be effective in detecting clinically significant macular edema as well as advanced stages of DR. Prior to the deep learning era, AI models have been trained based on “feature-based” learning to detect microaneurysms, hemorrhages, hard or soft exudates and vessel maps. RetmarkerDR is another system that has been used for DR screening, by screening out images without DR and sorting images that have DR into a group that requires further human grader assessment. A notable characteristic of RetmarkerDR is that it can track disease progression by comparing current images to those that were initially screened out, potentially providing insight on progression of DR.
In 2016, Abramoff et al reported a comparison of a deep-learning enhanced algorithm for the automated detection of DR to an identical algorithm that did not utilize deep learning and noted improved specificity for referable DR. This algorithm utilizes a previously reported reference consensus standard for referable DR, the International Clinical Classification of Diabetic Retinopathy, and provides four outputs: negative (none or mild DR), referable DR, vision threatening DR, and/or low exam quality which noted a limitation in analysis or image quality. Google also validated a DR detection algorithm which assigns an image a number between 0-1 that correlates with the likelihood of DR in the image. Studies have shown the sensitivity and specificity of this algorithm are above 90%. Ting et al described and validated a deep learning system that not only identifies vision threatening and referable DR, but also AMD and possible glaucoma in a multiethnic cohort.
In 2018, Abràmoff and colleagues obtained the first US Food and Drug Administration approval of their AI system IDx-DR system using deep learning, after a pivotal prospective clinical trial of 900 patients. All images were obtained using the Topcon Fundus camera (Topcon Medical). This deep learning platform functions by independently assessing the quality of the images inputted into the system and indicates whether there is “more than mild” DR present on imaging.Table 1 highlights a select group of AI algorithms for DR screening, given that there are many that have been described in the recent years.
In 2020, the EyeArt system by Eyenuk became another autonomous AI system that is FDA approved for the detection of mtmDR.  This system utilizes two 45-degrees fundus photos per eye and the cloud based software results in a report for the presence or absence of mtmDR or vision threatening DR (vtDR) within 60 seconds. A multicenter clinical trial comparing the AI system to a reading center grading of widefield images from the same eyes showed high sensitivity and specificity: sensitivity was 95.5% and specificity was 85% for detection of mtmDR; sensitivity was 95.1% and specificity was 89% for detection of vtDR without dilation. Imageability was high without dilation. A recent study has shown the EyeArt system has significantly higher sensitivity than general ophthalmologists or retina specialists for detection of mtmDR. 
AI programs have also utilized fundus photography via smartphones. For example, EyeNuk is an AI screening program that utilizes the EyeArt software which has shown positive results with over 95% sensitivity when using fundus images obtained from smartphones. EyeArt also utilizes standard desktop fundus cameras. As AI-enhanced screening tools become more widely utilized, it is also important to integrate screening for other diseases.
Retinopathy of Prematurity
Challenges in retinopathy of prematurity screening and diagnosis include diagnostic variability, paucity of specialists in ROP, and access to care issues. Machine learning systems have been developed as a possible solution to address these challenges. These systems evaluate vascular changes, features of the disease such as zone or stage, category of disease, and disease progression. Recent advancements in AI have allowed for the development of quantitative scores for defining features of disease, such as rating the degree of plus disease or identifying the number of regions with neovascularization. One such example is the i-ROP score, developed by the Imaging and Informatics in Retinopathy of Prematurity (i-ROP) Research Consortium. This was shown to be non-inferior to human expert diagnosis, notably when identifying vascular changes such as pre-plus and plus disease.
Additional computerized algorithms using automated methods for detection of plus disease have been previously reported, including ROPtool, Retinal Image multiScale Analysis (RISA), VesselMap, and Computer-Assisted Image Analysis of the Retina (CAIAR).The hallmark features of plus disease in ROP are defined based on the degree of arteriolar tortuosity and venular dilation. Both the ROPtool and the RISA systems perform exact measurements of vessel curvature relative to vessel length, and the ROPtool and the VesselMap define the location of vascular changes in relation to the optic disc. The VesselMap, unlike the former two computing modalities, measures the dimensions of the vasculature by evaluating the intensity of light seen in planes orthogonal to the long axis of vessels. The CAIAR system quantifies the extent of arteriolar tortuosity and venule width. Over time, these AI systems have been updated and expanded, and some have shown promising results with their demonstration of high sensitivity and specificity for the detection of different aspects of plus disease. Table 2 summarizes systems designed for automated detection of plus disease in retinopathy of prematurity.
AI algorithms have been applied to both fundus photographs and optical coherence tomography (OCT) to detect age-related macular degeneration . Using deep learning, AI was shown to achieve comparable diagnostic performance in detecting referable AMD when compared to human graders. In Burlina et al, the AI system was developed using AlexNet and more than 100,000 Age-Related Eye Disease Study (AREDS) stereoscopic fundus photos and demonstrated that the 4-step grading was comparable to that of a human grader. The AI system was further developed to estimate the 5-year risk of progression to advanced AMD. Subsequently, Grassman et al demonstrated that multiple convolutional neural networks can be trained on the same dataset and were able to outperform human graders.
Using retina OCT images, AI systems can be trained to perform segmentation, classification and prediction. In segmentation, many AI systems were shown to display highly accuracy in segmenting different retinal layers on OCT, and this is important to quantify intraretinal fluid, subretinal fluid and pigment epithelial detachment. When compared to non-computerized segmentation techniques, the deep learning algorithm developed by Lee et al properly differentiated fluid accumulation from other abnormal retinal findings, which was also demonstrated by the validated ReLayNet deep learning system. Volumetric analysis of intraretinal fluid and drusen have also been evaluated using deep learning methods.  For classification tasks, Kermany et al showed the value of using transfer learning in detecting three retinal conditions, namely drusen, neovascular AMD and diabetic macular edema.Following this study, De Fauw et al further confirmed not only the ability of deep learning in detecting >50 retinal conditions, but also the robustness of the AI system to help triage the urgency of referrals for patients with retinal diseases. Lastly, for the prediction tasks, using HARBOR data, Bogunovic et al and Erfurth-Schmidth demonstrated the ability of AI system to predict the long term visual outcome (at month 12) by integrating the OCT images segmented using deep learning, and utilized the machine learning approach with other clinical data for analysis. Table 3 summarizes current AI algorithms utilized in AMD.
FDA Approval of AI Algorithms
Software as a medical device (SaMD) is an area of rapidly expanding research and development. The FDA approval of IDx-DR in 2018 sparked complex discussions regarding future approval of independent AI algorithms. Subsequently, the FDA published a statement in April of 2019 proposing a novel regulatory framework for AI based medical devices. The guidelines for medical devices prior to this statement did not take into account the dynamic nature of machine learning algorithms. Previously, medical devices reviewed by the FDA were primarily static in nature and were evaluated by comparing them to similar products already on the market. It is also important to assess the quality of the training and validation datasets to ensure that the results are generalizable across diverse patient populations. The figure below describes a potential workflow for the FDA approval of AI systems.
The dynamic nature of AI systems makes it difficult to predict at what point the algorithms should be reviewed. While the initial training and validation datasets may yield promising results, it is possible that variations in output can occur as more data is processed by the system when it is available on the market. To address this issue, the FDA is implementing a system called pre-certification, which focuses on how the product and AI system is developed. The FDA also described categories of software modifications that may require a premarket submission, especially when considering changes that may introduce a new risk, prevent harm, or changes that significantly affect the clinical functionality of the software. The International Medical Device Regulators Forum (IMDRF) created a risk-based approach to categorize SaMD, ranging from I to IV, which reflects the risk associated with the clinical situation and device use. The FDA also described a “total product lifecycle regulatory approach” with regards to SaMD given that these products can adapt and improve with real-world use. This would enable the evaluation and monitoring of the software product from its premarket development to post-market performance. Regulatory guidelines specifically for AI is a step in the right direction as it allows AI based medical devices to evolve in ways to improve itself while also meeting the FDA’s standards for safety and efficacy.
AI retinal screening tools can have a limited scope in identifying a specific disease process such as DR but fail to comment or identify other pathology that could be clinically significant. For example, the IDx-DR system and the EyeArt system were created to identify DR and do not diagnose or screen for other pathology although many patients have co-existing ocular morbidities. However, as AI-enhanced screening tools become more widely utilized, it is also important to integrate screening for multiple disease processes. Ting et al described and validated a deep learning system that not only identifies vision threatening and referable DR, but also AMD and possible glaucoma in a multiethnic cohort.
Furthermore, it is important to consider general limitations of AI algorithms, including the “garbage in, garbage out” phenomenon, whereby the quality of inputs can limit the quality of the outputs, and the “black box dilemma,” in which the inherent lack of transparency in the algorithm may cause a certain level of distrust. It is important to assess whether the training sets include images that are generalizable to a multiethnic population and obtained using a variety of commercially available imaging devices. If a deep learning algorithm is applied only using a specific imaging device, it may limit widespread application. Additionally, costs associated with the development of large training and validation datasets can be significant and represent an area for future development.
Retinal image analysis using AI has already demonstrated significant potential for expanded screening, particularly in rural areas. Moorfields Eye Hospital in collaboration with DeepMind have designed a deep learning system with a multifold mechanism of action: (1) OCT images are assessed through a segmentation algorithm for the presence of any features that may indicate underlying pathology, (2) based on these findings, the image is assigned to its corresponding retinal disease, and (3) the current state of the disease is rated as either urgent, semi-urgent, routine, or observation. This method has shown to be non-inferior to expert diagnosis and allows for timely referral of cases to the ophthalmic specialist. Such a system may ultimately have a role in currently established tele-ophthalmology programs and other virtual clinics that are cost-effective approach to maximizing outreach to distant communities. Furthermore, we are likely to see more FDA approved AI systems for not only DR but other disease process as well. Ultimately, implementing an automated system into primary care, endocrinology, and community health centers can improve screening for retinal pathology and facilitate the decision to refer to ophthalmology for further management.
- Kapoor R, Walters SP, Al-Aswad LA. The current state of artificial intelligence in ophthalmology. Survey of Ophthalmology. 2019;64(2):233-240.
- Roach, L. “Artificial Intelligence.” EyeNet Magazine, Nov. 2017, www.aao.org/eyenet/article/artificial-intelligence.
- The ultimate guide to AI in radiology. The Ultimate Guide to AI in Radiology. https://www.quantib.com/the-ultimate-guide-to-ai-in-radiology. Accessed November 13, 2019.
- Cole, Emily D., et al. "The definition, rationale, and effects of thresholding in OCT angiography." Ophthalmology Retina 1.5 (2017): 435-447.
- Alam M, Le D, Lim JI, Chan RVP, Yao X. Supervised Machine Learning Based Multi-Task Artificial Intelligence Classification of Retinopathies. Journal of Clinical Medicine. 2019;8(6):872. doi:10.3390/jcm8060872.
- Abràmoff MD, Lou Y, Erginay A, Clarida W, Amelon R, Folk JC, et al. Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Invest Ophthalmol Vis Sci. 2016;57:5200–6.
- Van der Heijden AA, Abramoff MD, Verbraak F, van Hecke MV, Liem A, Nijpels G. Validation of automated screening for refer-able diabetic retinopathy with the IDx‐DR device in the Hoorn Diabetes Care System. Acta Ophthalmol (Copenh). 2018;96:63–8.
- Gulshan V, Peng L, Coram M, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 2016;316:2402–10.
- Ting DSW, Pasquale LR, Peng L, et al. Artificial intelligence and deep learning in ophthalmology. British Journal of Ophthalmology. 2019;103(2):167-175.
- Ribeiro L, OliveiraCM, Neves C, Ramos JD, Ferreira H, Cunha- Vaz J. Screening for diabetic retinopathy in the central region of Portugal. Added value of automated ‘disease/no disease’ grading. Ophthalmologica. 2015;233:96–103.
- Leicht, Simon F., et al. "Microaneurysm turnover in diabetic retinopathy assessed by automated RetmarkerDR image analysis-potential role as biomarker of response to ranibizumab treatment." Ophthalmologica 231.4 (2014): 198-203.
- Raumviboonsuk P, Krause J, Chotcomwongse P, Sayres R, Raman R, Widner K, et al. Deep learning versus human graders for classifying diabetic retinopathy severity in a nationwide screening program. npj Digit Med. 2019;2:25.
- Ting DSW, Cheung CY-L, Lim G, et al. Development and Validation of a Deep Learning System for Diabetic Retinopathy and Related Eye Diseases Using Retinal Images From Multiethnic Populations With Diabetes. JAMA. 2017;318(22):2211. doi:10.1001/jama.2017.18152.
- Abramoff MD, Lavin PT, Birch M, et al. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digit Med 2018;1:1–8.
- Tufail A, Kapetanakis VV, Salas-Vega S, Egan C, Rudisill C, Owen CG, et al. An observational study to assess if automated diabetic retinopathy image assessment software can replace one or more steps of manual imaging grading and to determine their cost- effectiveness. Health Technol Assess (Rockv).2016;20:1–72. xxviii
- Rajalakshmi R, Subashini R, Anjana RM, Mohan V. Automated diabetic retinopathy detection in smartphone-based fundus photography using artificial intelligence. Eye 2018;32:1138.
- Bhaskaranand M, Ramachandra C, Bhat S, Cuadros J, Nittala MG, Sadda S, et al. Automated Diabetic Retinopathy Screening and Monitoring Using Retinal Fundus Image Analysis. J Diabetes Sci Technol 2016;10:254-61.
- Ipp E, Liljenquist D, Bode B, Shah VN, Silverstein S, Regillo CD, Lim JI, Sadda S, Domalpally A, Gray G, Bhaskaranand M, Ramachandra C, Solanki K; EyeArt Study Group. Pivotal Evaluation of an Artificial Intelligence System for Autonomous Detection of Referrable and Vision-Threatening Diabetic Retinopathy. JAMA Netw Open. 2021 Nov 1;4(11):e2134254. Erratum in: JAMA Netw Open. 2021 Dec 1;4(12):e2144317. PMID: 34779843; PMCID: PMC8593763.
- Lim JI, Regillo CD, Sadda SR, Ipp E, Bhaskaranand M, Ramachandra C, Solanki K, EyeArt Study Sugbroup. Artificial Intelligence Detection of Diabetic Retinopathy: Subgroup Comparison of the EyeArt System with Ophthalmologists' Dilated Exams . Ophthalmology Science 2022; In Press.
- Ting DSW, Lee AY, Wong TY. An Ophthalmologist's Guide to Deciphering Studies in Artificial Intelligence. Ophthalmology. 2019;126(11):1475-1479.
- Ataer-Cansizoglu E, Bolon- Canedo V, Campbelll JP, et al. Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity: performance of the ‘‘i-ROP’’ system and image features associated with expert diagnosis. TransVis Sci Tech. 2015; 4(6):5, doi:10.1167/tvst.4.6.5.
- Xiao S, Bucher F, Wu Y, et al. Fully automated, deep learning segmentation of oxygen-induced retinopathy images. JCI Insight. 2017;2(24):e97585. Published 2017 Dec 21. doi:10.1172/jci.insight.97585.
- Taylor S, Brown JM, Gupta K, et al. Monitoring disease progression with a quantitative severity scale for retinopathy of prematurity using deep learning. JAMA Ophthalmol 2019.
- Brown JM, Campbell JP, Beers A, et al. Automated diagnosis of plus disease in retinopathy of prematurity using deep convolutional neural networks. JAMA Ophthalmol 2018;136:803–10.
- Wittenberg LA, Jonsson NJ, Chan RV, Chiang MF. Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity. J Pediatr Ophthalmol Strabismus. 2012;49(1):11–20. doi:10.3928/01913913-20110222-01
- Wallace DK, Freedman SF, Zhao Z, Jung S. Accuracy of ROPtool vs Individual Examiners in Assessing Retinal Vascular Tortuosity. Arch Ophthalmol. 2007;125(11):1523–1530.
- Swanson C, Cocker KD, Parker KH, Moseley MJ, Fielder AR. Semiautomated computer analysis of vessel growth in preterm infants without and with ROP. Br J Ophthalmol. 2003;87(12):1474–1477. doi:10.1136/bjo.87.12.1474
- Wang J, Ju R, Chen Y, et al. Automated retinopathy of prematurity screening using deep neural networks. EBioMedicine 2018; 35:361 – 368.
- Redd TK, Campbell JP, Brown JM, et al. Evaluation of a deep learning image & assessment system for detecting severe retinopathy of prematurity. Br J Ophthalmol 2018; 2018-313156.
- Rani P, Elagiri Ramalingam R, Rajamani KT, et al. Multiple instance learning: robust validation on retinopathy of prematurity. Int J Ctrl Theory Appl 2016; 9:451 – 459.
- Brown JM, Campbell JP, Beers A, et al. Automated diagnosis of plus disease in retinopathy of prematurity using deep convolutional neural networks. JAMA Ophthalmol 2018; 136:803–810.
- Worrall DE, Wilson CM, Brostow GJ. Automated Retinopathy of Prematurity Case Detection with Convolutional Neural Networks. Deep Learning and Data Labeling for Medical Applications Lecture Notes in Computer Science. 2016:68-76. doi:10.1007/978-3-319-46976-8_8.
- Bolon-Canedoa V, Ataer-Cansizoglub E, Erdogmusb D, et al. Dealing with inter-expert variability in retinopathy of prematurity: a machine learning approach. Comput Methods Programs Biomed 2015; 122:1–15.
- Wallace DK, Zhao Z, Freedman SF. A pilot study using ‘ROPtool’ to quantify plus disease in retinopathy of prematurity. J Am Assoc Pediatr Ophthalmol Strabismus 2007; 11:381 – 387.
- Gelman R, Jiang L, Du YE, et al. Plus disease in retinopathy of prematurity: pilot study of computer-based and expert diagnosis. J Am Assoc Pediatr Ophthalmol Strabismus 2007; 11:532 – 540.
- Burlina PM, Joshi N, Pekala M, Pacheco KD, Freund DE, Bressler NM. Automated grading of age-related macular degeneration from color fundus images using deep convolutional neural networks. JAMA Ophthalmol 2017;135:1170‐6.
- Grassmann F, Mengelkamp J, Brandl C, Harsch S, Zimmermann ME, Linkohr B, et al. A deep learning algorithm for prediction of age-related eye disease study severity scale for age-related macular degeneration from color fundus photography. Ophthalmology 2018;125:1410-20.
- Akkara J, Kuriakose A. Role of artificial intelligence and machine learning in ophthalmology. Kerala Journal of Ophthalmology. 2019;31(2):150-160.
- Lee CS, Tyring AJ, Deruyter NP, Wu Y, Rokem A, Lee AY. Deep-learning based, automated segmentation of macular edema in optical coherence tomography. Biomed Opt Express. 2017;8(7):3440–3448. Published 2017 Jun 23. doi:10.1364/BOE.8.003440
- Roy AG, Conjeti S, Karri SPK, et al.ReLayNet: retinal layer and fluid segmentation of macular optical coherence tomography using fully convolutional networks. Biomed Opt Express. 2017;8(8):3627-3642.
- Schlegl T, Waldstein SM, Bogunovic H, et al. Fully Automated Detection and Quantification of Macular Fluid in OCT Using Deep Learning. Ophthalmology. 2018;125(4):549e58
- Venhuizen FG, van Ginneken B, Liefers B, et al. Deep learning approach for the detection and quantification of intraretinal cystoid fluid in multivendor optical coherence tomography. Biomed Opt Express. 2018;9(4):1545-1569.
- Kermany DS, Goldbaum M, Cai W, et al. Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning. Cell. 2018;172(5). doi:10.1016/j.cell.2018.02.010.
- Fauw JD, Ledsam JR, Romera-Paredes B, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nature Medicine. 2018;24(9):1342-1350. doi:10.1038/s41591-018-0107-6.
- Bogunovic H, Waldstein SM, Schlegl T, et al.Prediction of Anti-VEGF Treatment Requirements in Neovascular AMD Using a Machine Learning Approach. Invest Ophthalmol Vis Sci. 2017;58(7):3240e8.
- Rohm M, Tresp V, Muller M, et al. Predicting Visual Acuity by Using Machine Learning in Patients Treated for Neovascular Age-Related Macular Degeneration. Ophthalmology. 2018;125(7):1028e36
- Treder M, Lauermann JL, Eter N. Automated detection of exudative age-related macular degeneration in spectral domain optical coherence tomography using deep learning. Graefes Arch Clin Exp Ophthalmol 2018;256:259–65. 10.1007/s00417-017-3850-3
- Artificial Intelligence and Machine Learning in Software.U.S. Food and Drug Administration. https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device. Accessed November 12, 2019.
- Keane PA, Topol EJ. With an eye to AI and autonomous diagnosis. npj Digital Medicine. 2018;1(1). doi:10.1038/s41746-018-0048-y.
- Korot E, Wood E, Weiner A, Sim DA, Trese M. A renaissance of teleophthalmology through artificial intelligence. Eye. 2019;33(6):861-863.