[1]陈铭海,白芳,陶海.多模态数据融合技术及其在眼科领域的应用研究进展[J].眼科新进展,2024,44(3):248-252.[doi:10.13389/j.cnki.rao.2024.0049]
 CHEN Minghai,BAI Fang,TAO Hai.Multimodal data fusion and research progress of its application in ophthalmology[J].Recent Advances in Ophthalmology,2024,44(3):248-252.[doi:10.13389/j.cnki.rao.2024.0049]
点击复制

多模态数据融合技术及其在眼科领域的应用研究进展/HTML
分享到:

《眼科新进展》[ISSN:1003-5141/CN:41-1105/R]

卷:
44卷
期数:
2024年3期
页码:
248-252
栏目:
文献综述
出版日期:
2024-03-05

文章信息/Info

Title:
Multimodal data fusion and research progress of its application in ophthalmology
作者:
陈铭海白芳陶海
100039 北京市,解放军总医院第三医学中心眼科医学部(陈铭海,白芳,陶海); 230032 安徽省合肥市,安徽医科大学第五临床学院(陈铭海,白芳,陶海)
Author(s):
CHEN Minghai12BAI Fang12TAO Hai12
1.Senior Department of Ophthalmology,the Third Medical Center of PLA General Hospital,Beijing 100039,China
2.The Fifth Clinical College of Anhui Medical University,Hefei 230032,Anhui Province,China
关键词:
人工智能多模态数据融合技术眼科学研究进展文献综述
Keywords:
artificial intelligence multimodal data fusion ophthalmology research progress review
分类号:
R770.4
DOI:
10.13389/j.cnki.rao.2024.0049
文献标志码:
A
摘要:
多模态数据融合技术(MDF)是人工智能领域中的一种数据处理技术,是指利用计算机对多模态数据,如图片、文字、音频与视频等进行综合处理的技术。与传统单模态数据处理相比,MDF对描述对象的分析有更高的准确性与可靠性。近年来,该技术在眼科领域的应用研究取得了较大进展,研究表明该技术能提高人工智能辅助诊断的准确性。本文综述了MDF的原理与优势、发展简况、架构与主要技术难点、MDF在眼科临床的应用现况以及目前存在的尚待解决的问题。
Abstract:
Multimodal data fusion (MDF), a data processing technology in the field of artificial intelligence, refers to a comprehensive processing technology of multimodal data such as pictures, texts, audio and videos by computer. Compared with traditional single-mode data processing, MDF has higher accuracy and reliability in the analysis of described objects. In recent years, the application of MDF in ophthalmology has made great progress, and studies have shown that this technology can improve the accuracy of artificial intelligence-assisted diagnosis. This paper review the principles and advantages, brief history of development, architecture and technical difficulties of MDF, as well as its application in ophthalmologic clinics and problems to be solved.

参考文献/References:

[1] TING D S W,PASQUALE L R,PENG L,CAMPBELL J P,LEE A Y,RAMAN R,et al.Artificial intelligence and deep learning in ophthalmology[J].Br J Ophthalmol,2019,103(2):167-175.
[2] 陈健祺,李睿扬,林铎儒,项毅帆,赖伟翊,林浩添.人工智能在眼病筛查和诊断中的研究进展[J].眼科学报,2022,37(3):208-213.
CHEN J Q,LI R Y,LIN D R,XIANG Y F,LAI W Y,LIN H T.Research progress of artificial intelligence in screening and diagnosis of eye diseases[J].Eye Sci,2022,37(3):208-213.
[3] 何俊,张彩庆,李小珍,张德海.面向深度学习的多模态融合技术研究综述[J].计算机工程,2020,46(5):1-11.
HE J,ZHANG C Q,LI X Z,ZHANG D H.Survey of research on multimodal fusion technology for deep learning [J].Comput Eng,2020,46(5):1-11.
[4] BALTRUSAITIS T,AHUJA C,MORENCY L P.Multimodal machine learning:a survey and taxonomy[J].IEEE Trans Pattern Anal Mach Intell,2019,41(2):423-443.
[5] YUHAS B P,GOLDSTEIN M H,SEJNOWSKI T J.Integration of acoustic and visual speech signals using neural networks[J].IEEE Commun Mag,1989,27(11):65-71.
[6] MEYER C R,BOES J L,KIM B,BLAND P H,ZASADNY K R,KISON P V,et al.Demonstration of accuracy and clinical versatility of mutual information for automatic multimodality image fusion using affine and thin-plate spline warped geometric deformations[J].Med Image Anal,1997,1(3):195-206.
[7] KLINE A,WANG H,LI Y,DENNIS S,HUTCH M,XU Z X,et al.Multimodal machine learning in precision health:a scoping review[J].NPJ Digit Med,2022,5(1):171.
[8] 任泽裕,王振超,柯尊旺,李哲,吾守尔·斯拉木.多模态数据融合综述[J].计算机工程与应用,2021,57 (18):49-64.
REN Z Y,WANG Z C,KE Z W,LI Z,WUSHOUR S.Survey of multimodal data fusion[J].Comput Eng Appl,2021,57 (18):49-64.
[9] ANTON N,DOROFTEI B,CURTEANU S,CATALIN L,LLIE O D,TARCOVEANU F,et al.Comprehensive review on the use of artificial intelligence in ophthalmology and future research directions[J].Compute Diagnostics (Basel),2022,13(1):100.
[10] 李朝辉,吴畏.重视人工智能在眼科领域的应用[J].解放军医学院学报,2022,43(10):1010-1013.
LI Z H,WU W.Prospects of artificial intelligence in ophthalmology[J].Acad J Chin PLA Med Sch Jan,2022,43(10):1010-1013.
[11] WONG W L,SU X,LI X,CHEUNG C M G,KLEIN R,CHENG CY,et al.Global prevalence of age-related macular degeneration and disease burden projection for 2020 and 2040:a systematic review and meta-analysis[J].Lancet Glob Health,2014,2(2):e106-e116.
[12] JIN K,YAN Y,CHEN M L,WANG J,PAN X J,LIU X D,et al.Multimodal deep learning with feature level fusion for identification of choroidal neovascularization activity in age-related macular degeneration[J].Acta Ophthalmol,2022,100(2):e512-e520.
[13] YOO T K,CHOI J Y,SEO J G,RAMASUBRAMANIAN B,SELVAPERUMAL S,KIM D W.The possibility of the combination of OCT and fundus images for improving the diagnostic accuracy of deep learning for age-related macular degeneration:a preliminary experiment[J].Med Biol Eng Comput,2019,57(3):677-687.
[14] IACOVACCI J,PALORINI F,CICCHETTI A,FIOTINO C,RANCATI T.Dependence of the AUC of NTCP models on the observational dose-range highlights cautions in comparison of discriminative performance[J].Phys Med,2023,113:102654.
[15] WANG W S,LI X R,XU Z Y,YU W H,ZHAO J C,DING D Y,et al.Learning two-stream CNN for multi-modal age-related macular degeneration categorization[J].IEEE J Biomed Health Inform,2022,26(8):4111-4122.
[16] CHEN M L,JIN K,YAN Y,LIU X D,HUANG X L,GAO Z Y,et al.Automated diagnosis of age-related macular degeneration using multi-modal vertical plane feature fusion via deep learning[J].Med Phys,2022,49(4):2324-2333.
[17] HUANG C,XIE K R,ZHANG Y H,LI M C,YUAN S T,CHEN Q.Choroidal neovascularization fusion visualization for spectral-domain optical coherence tomography and optical coherence tomography angiography images[J].Med Phys,2021,48(4):1731-1738.
[18] HSU M Y,CHIOU J Y,LIU J T,LEE C M,LEE Y W,CHOU C C,et al.Deep learning for automated diabetic retinopathy screening fused with heterogeneous data from EHRs can lead to earlier referral decisions[J].Transl Vis Sci Technol,2021,10(9):18.
[19] TSENG V S,CHEN C L,LIANG C M,TAI M C,LIU J T,WU P Y,et al.Leveraging multimodal deep learning architecture with retina lesion information to detect diabetic retinopathy[J].Transl Vis Sci Technol,2020,9(2):41.
[20] HE X X,DENG Y,FANG L Y,PENG Q H.Multi-modal retinal image classification with modality-specific attention network[J].IEEE Trans Med Imaging,2021,40(6):1591-1602.
[21] AI Z,HUANG X,FENG J,WANG H,TAO Y,ZENG F X,et al.FN-OCT:Disease detection algorithm for retinal optical coherence tomography based on a fusion network[J].Front Neuroinform,2022,16:876927.
[22] KERMANY D S,GOLDBAUM M,CAI W J,VALENTIM C C S,LIANG H Y,BAXTER S L,et al.Identifying medical diagnoses and treatable diseases by image-based deep learning [J].Cell,2018,172(5):1122-1131.
[23] WEN D J,YU Z H,YANG Z W,ZHENG C Z,REN X J,SHAO Y,et al.Deep learning-based postoperative visual acuity prediction in idiopathic epiretinal membrane[J].BMC Ophthalmol,2023,23(1):361.
[24] KOLAR R,KUBECKA L,JAN J.Registration and fusion of the autofluorescent and infrared retinal images[J].Int J Biomed Imaging,2008,2008:513411-513478.
[25] YI S L,ZHANG G,QIAN C X,LU Y Q,ZHONG H,HE J F.A multimodal classification architecture for the severity diagnosis of glaucoma based on deep learning[J].Front Neurosci,2022,16.
[26] XIONG J,LI F,SONG D P,TANG G X,HE J J,GAO K,et al.Multimodal machine learning using visual fields and peripapillary circular OCT scans in detection of glaucomatous optic neuropathy[J].Ophthalmology,2022,129(2):171-180.
[27] KIHARA Y,MONTESANO G,CHEN A,AMERASINGHE N,DIMITRIOU C,JACOB A,et al.Policy-driven,multimodal deep learning for predicting visual fields from the optic disc and OCT imaging[J].Ophthalmology,2022,129(7):781-791.
[28] De ZANET S I,CILLER C,RUDOLPH T,MAEDER P,MUNIER F,BALMER A,et al.Landmark detection for fusion of fundus and MRI toward a patient-specific multimodal eye model[J].IEEE Trans Biomed Eng,2015,62(2):532-540.

相似文献/References:

[1]赵金凤,吴桢泉,郑磊,等.人工智能在眼科疾病诊疗的应用现状[J].眼科新进展,2019,39(5):495.[doi:10.13389/j.cnki.rao.2019.0114]
 ZHAO Jin-Feng,WU Zhen-Quan,ZHENG Lei,et al.Application of artificial intelligence in the diagnosis and treatment of ocular diseases[J].Recent Advances in Ophthalmology,2019,39(3):495.[doi:10.13389/j.cnki.rao.2019.0114]
[2]余燕,侯银芬,吴昌凡,等.人工智能深度学习技术在常见眼病辅助诊断的应用现状和进展[J].眼科新进展,2020,40(4):396.[doi:10.13389/j.cnki.rao.2020.0092]
 YU Yan,HOU Yinfen,WU Changfan,et al.Application and progress of artificial intelligence deep learning technology in diagnosis of common eye diseases[J].Recent Advances in Ophthalmology,2020,40(3):396.[doi:10.13389/j.cnki.rao.2020.0092]
[3]孙铁,张雨晴,邵毅.人工智能及其在眼科疾病诊疗中的应用[J].眼科新进展,2020,40(8):793.[doi:10.13389/j.cnki.rao.2020.0181]
 SUN Tie,ZHANG Yuqing,SHAO Yi.The application of artificial intelligence on ophthalmic diseases[J].Recent Advances in Ophthalmology,2020,40(3):793.[doi:10.13389/j.cnki.rao.2020.0181]
[4]郑武,阮坤炜,吴天添,等.人工智能糖尿病视网膜病变筛查系统与眼科医师诊断结果的一致性分析[J].眼科新进展,2020,40(12):1170.[doi:10.13389/j.cnki.rao.2020.0260]
 ZHENG Wu,RUAN Kunwei,WU Tiantian,et al.Consistency of artificial intelligence screening system with ophthalmologist for diagnosing of diabetic retinopathy[J].Recent Advances in Ophthalmology,2020,40(3):1170.[doi:10.13389/j.cnki.rao.2020.0260]
[5]《人工智能在干眼临床诊断中的应用专家共识(0)》专家组,中国医药教育协会眼科影像与智能医疗分会,中国人口文化促进会角膜病与眼表疾病分会.人工智能在干眼临床诊断中的应用专家共识(2023)[J].眼科新进展,2023,43(4):253.[doi:10.13389/j.cnki.rao.2023.0052]
 Expert Consensus on Clinical Application of Artificial Intelligence in Dry Eyes (0) Expert Group,Ophthalmic Imaging and Intelligent Medicine Branch of China Medical Education Association,Corneal and Ocular Surface Diseases Branch of Chinese Population and Society Promotion Association.Expert consensus on clinical application of artificial intelligence in dry eyes (2023)[J].Recent Advances in Ophthalmology,2023,43(3):253.[doi:10.13389/j.cnki.rao.2023.0052]
[6]杨丽丹,李青蒨,陈倩茵,等.人工智能在青光眼诊断中的研究进展[J].眼科新进展,2023,43(6):500.[doi:10.13389/j.cnki.rao.2023.0102]
 YANG Lidan,LI Qingqian,CHEN Qianyin,et al.Research progress of artificial intelligence in the diagnosis of glaucoma[J].Recent Advances in Ophthalmology,2023,43(3):500.[doi:10.13389/j.cnki.rao.2023.0102]
[7]《人工智能辅助诊断早产儿视网膜病变应用指南(0)》专家组,中国医药教育协会眼科影像与智能医疗分会,国际转化医学会眼科专委会,等.人工智能辅助诊断早产儿视网膜病变应用指南(2023)[J].眼科新进展,2023,43(9):673.[doi:10.13389/j.cnki.rao.2023.0136]
 Expert Workgroup of Guidelines for the Application of Artificial Intelligence in Retinopathy of Premature Infants0,Ophthalmic Imaging and Intelligent Medicine Branch of Chinese Medicine Education Association,Ophthalmology Committee of International Association of Translational Medicine,et al.Guidelines for the application of artificial intelligence in the auxiliary diagnosis of retinopathy of prematurity (2023)[J].Recent Advances in Ophthalmology,2023,43(3):673.[doi:10.13389/j.cnki.rao.2023.0136]
[8]《人工智能在甲状腺相关性眼病中的应用指南(0)》专家组,中国医药教育协会眼科影像与智能医疗分会,中国医师协会眼科分会眼肿瘤专委会,等.人工智能在甲状腺相关性眼病中的应用指南(2023)[J].眼科新进展,2023,43(11):841.[doi:10.13389/j.cnki.rao.2023.0169]
 Expert Workgroup of Guidelines for the Application of Artificial Intelligence in Thyroid Associated Ophthalmopathy(0),Ophthalmic Imaging and Intelligent Medicine Branch of Chinese Medicine Education Association,Ocular Oncology Committee of the Ophthalmology Branch of the Chinese Medical Doctor Association,et al.Guidelines for the application of artificial intelligence in thyroid-associated ophthalmopathy (2023)[J].Recent Advances in Ophthalmology,2023,43(3):841.[doi:10.13389/j.cnki.rao.2023.0169]
[9]《人工智能在神经眼科疾病诊断中的应用指南(0)》专家组,中国医药教育协会眼科影像与智能医疗分会,国际转化医学会眼科专业委员会,等.人工智能在神经眼科疾病诊断中的应用指南(2023)[J].眼科新进展,2023,43(12):925.[doi:10.13389/j.cnki.rao.2023.0185]
 Expert Workgroup of Guidelines for Application of Artificial Intelligence in the diagnosis of neuro-ophthalmic diseases (0),Ophthalmic Imaging and Intelligent Medicine Branch of Chinese Medicine Education Association,Ophthalmology Committee of International Association of Translational Medicine,et al.Guidelines for the application of artificial intelligence in the diagnosis of neuro-ophthalmic diseases[J].Recent Advances in Ophthalmology,2023,43(3):925.[doi:10.13389/j.cnki.rao.2023.0185]
[10]任章军,余进海,桑泽曦,等.人工智能深度学习在眼眶病及眼肿瘤疾病诊疗中的应用研究现状[J].眼科新进展,2024,44(2):163.[doi:10.13389/j.cnki.rao.2024.0032]
 REN Zhangjun,YU Jinhai,SANG Zexi,et al.Research advancement of the application of artificial intelligence deep learning in the diagnosis and treatment of orbital diseases and ocular tumors[J].Recent Advances in Ophthalmology,2024,44(3):163.[doi:10.13389/j.cnki.rao.2024.0032]

备注/Memo

备注/Memo:
军队后勤科研重点项目(编号:BLB23C003)
更新日期/Last Update: 2024-03-05