Frontiers of Data and Computing ›› 2020, Vol. 2 ›› Issue (2): 91-100.

doi: 10.11871/jfdc.issn.2096-742X.2020.02.007

Special Issue: “数据分析技术与应用”专刊

• Special Issue: Data Analysis Technology & Application • Previous Articles     Next Articles

Research on Geoscience Big Data Processing Framework and Key Techniques

Zhang Yaonan1,2,3,*(),Ai Minghao1,2,Kang Jianfang1,2,3,Min Yufang1,2   

  1. 1. Northwest Institute of Eco-Environment and Resources, Chinese Academy of Sciences, Lanzhou, Gansu 730000, China
    2. National cryosphere Desert Scientific Data Center, Lanzhou, Gansu 730000, China
    3. Gansu Data Engineering ang Technology Research Center for Resource and Environment, Lanzhou, Gansu 73000, China
  • Received:2020-01-16 Online:2020-04-20 Published:2020-06-03
  • Contact: Yaonan Zhang E-mail:yaonan@lzb.ac.cn

RichHTML

42

PDF

388

Abstract:

[Objective] As one of the special data science methods, big data brings great opportunities for geological research. Meanwhile, the characteristics of geological data such as multi-source heterogeneity, spatial-temporal correlation, multi-scale and uncertainty bring great challenges for the data processing. [Methods] On the basis of a detailed analysis of aforementioned characteristics, this study proposes a geological data processing framework to solve problems of multi-source data integration and heterogeneous data synthesis in geoscience field combined with a variety of big data technologies like data association, middleware systems, micro services and container technique. Besides, geological models are embedded in this framework in order to improve the expertness of data process. [Results] The framework and its key technologies have been applied in the construction of the National Glacier and Frozen Soil Scientific Data Center, the disaster datasets for the China-Pakistan Corridor as well as the High and Cold Environment United Observation Cloud. [Conclusions] This study is expected to broaden the data processing dimension and support multi-theme, multi-scale research and knowledge discovery in geoscience. In future, it will be adapted to the processing of geological data from a wider range of sources such as the internet, social networks, and printed media. The integration of artificial intelligence technologies will enable the framework to provide smarter and faster geological data processing results.

Key words: earth scientific big data, geological data processing methods, data convergence, heterogeneous data integration