Research on Multi-Source Heterogeneous Data Access Technology and Its Application Framework in Vertical Field

doi:10.11871/jfdc.issn.2096-742X.2024.05.015

Abstract

Abstract:

[Objective] The breakthrough of multi-source heterogeneous data access technology is particularly important for effectively solving the severe problem of "data island" in the vertical field, achieving global, authenticity, real-time guarantee of data acquisition, and data fusion applications. [Methods] Through literature analysis, current researches on multi-source heterogeneous data access technologies of ETL based, semantic ontology based, robot process automation based and cloud side integration resource reflection mechanism based are summarized. The characteristics of various technologies are analyzed. [Results] Typical frameworks for the application of four multi-source heterogeneous data access technologies in the vertical field are explored and designed, and four technical features are summarized and compared, in order to provide valuable demonstration and reference for relevant practical applications. [Conclusions] A theoretical basis for the integration and interoperability practices of big data in the vertical field is provided, and how to fully utilize key technical methods to better solve the technical limitations of real-time data access will be the focus of future research.

Key words: multi-source heterogeneous data, data access, interoperability technology

LIU Baoqi, XUE Fei, MENG Shengwei, ZHANG Chenglu. Research on Multi-Source Heterogeneous Data Access Technology and Its Application Framework in Vertical Field[J]. Frontiers of Data and Computing, 2024, 6(5): 159-168, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2024.05.015.

Figures/Tables 5

Fig.1

Fig.2

Fig.3

Fig.4

Table 1

References 18

[1]	SABTU A, FIRDAUS M A N, NUR A S N, et al. The challenges of Extract, Transform and Loading (ETL) system implementation for near real-time environment[C]. 2017 International Conference on Research and Innovation in Information Systems, 2017: 1-7.
[2]	MUDDASIR M, RAGHUVEER K. Study of Methods to Achieve Near Real Time ETL[C]. 2017 International Conference on Current Trends in Computer, Electrical, Electronics and Communication, 2017:436-441.
[3]	刘吉峰, 罗勐, 梁聪聪. 黄河多源异构凌情数据仓库初步研究[J]. 人民黄河, 2022, 44(2): 27-31.
[4]	张培, 黄智源, 陈琨, 等. 数字化车间多源异构质量数据集成方案研究[J]. 现代制造工程, 2015(1): 59-65.
[5]	STUDER R, BENJAMINS V R, FENSEL D. Knowledge engineering: principles and methods[J]. Data & Knowledge Engineering, 1998, 25(1-2), 161-197.
[6]	刘金晶, 曹文浩. 基于本体论的多源异构数据集成方法研究[J]. 软件导刊, 2018, 17(3): 198-200.
[7]	李跃先, 殷传涛, 魏亿钢. 基于本体与中间件的科技资源数据集成方法[J]. 标准科学, 2021(5): 21-28.
[8]	ABHIJIT A, BISWANATH D, ANIMESH D. Finding most informative common ancestor in cross-ontological semantic similarity assessment: An intrinsic information content-based approach[J]. Expert Systems with Applications, 2022(19): 1-14.
[9]	SEQUEDA F, MIRANKER P. A Pay-As-You-Go Methodology for Ontology-Based Data Access[J]. IEEE Internet Computing, 2017, 21(2): 92-96.
[10]	朱勤斯, 虞慧群. 一种基于语义网技术和本体的数据集成方法[J]. 华东理工大学学报(自然科学版), 2009, 35(1): 119-124.
[11]	叶霞, 刘睿珩, 许飞翔, 等. 基于SOA和本体的指挥信息系统数据集成研究[J]. 火力与指挥控制, 2022, 47(3): 136-143.
[12]	SIMONE A, MARCO L, ANDREA M, et al. Reactive synthesis of software robots in RPA from user interface logs[J]. Computers in Industry, 2022(142): 1-22.
[13]	WEWERKA J, MICUS C, REICHERT M. Seven Guidelines for Designing the User Interface in Robotic Process Automation[C]. 2021 IEEE 25th International Enterprise Distributed Object Computing Workshop, 2021: 157-165.
[14]	任腾云, 戚珺, 金陶岚, 等. 国网江苏电力智慧财务体系的构建[J]. 财务与会计, 2022(4): 66-68.
[15]	程平. RPA审计机器人: 理论框架与研发策略[J]. 会计之友, 2021(19): 2-7.
[16]	程平, 邓湘煜. RPA财务数据分析机器人: 理论框架与研发策略[J]. 会计之友, 2022(13): 148-155.
[17]	群众新闻网. 破解信息孤岛的接口高效互操作技术与燕云DaaS系统发布[EB/OL]. [2022-09-26]. https://www.sxdaily.com.cn/2021-09/24/content_9242050.html.
[18]	高可信软件技术教育部重点实验室官网. “云-端融合系统的资源反射机制及高效互操作技术”获中国电子学会技术发明特等奖[EB/OL]. [2022-09-26]. http://hcst.pku.edu.cn/info/1029/1075.htm.

多源异构数据接入技术	基于ETL	基于语义本体	基于RPA	基于云—端融合资源反射机制
接入方式	侵入式	侵入式	非侵入式	非侵入式
语义推理	不需要	需要	不需要	需要
领域依赖性	低	高	低	低
易开发性	适中	较难	较易	较难
局限性	无法实时数据接入，且复杂高并发任务接入较慢	本体构建需要大量的专家经验与人工操作	无法处理需要理解复杂业务逻辑、进行复杂决策的任务	完全“黑盒”，数据关系难获取，数据建模可解释性差
关键方法	ETL实时数据抽取方法；增量数据ETL自动采接整编方法；基于多线程模型的分布式ETL方法；ETL集群方法	面向主题域的数据内容语义标签生成方法；本体语义映射方法；基于内容的数据约束关系自动发现方法；语义编排与规则编译方法	基于RAP+AI的常规变体下自适应方法；大模型与RPA集成方法	基于逆向工程原理的数据访问接口智能生成方法；数据抽象建模方法；兼容DOA数字标识的数据溯源方法；基于计算反射的智能合约引擎