A Lightweight Remote Sensing Data Distributed Scheduling Framework DataboxMR

doi:10.11871/jfdc.issn.2096-742X.2021.03.009

Abstract

Abstract:

[Objective] At present, processing remote sensing data based on a common platform not only improves processing efficiency, but also introduces problems such as low efficiency in executing lightweight tasks, high threshold for end users, and high migration costs. This paper aims to reduce the complexity faced by users in handling lightweight tasks as well as the migration cost caused by common platform processing, and improve the user's control over the task scheduling. [Methods] A lightweight distributed scheduling framework DataboxMR is proposed for efficient processing of remote sensing data. The framework is based on UDF (User-Defined Function) technology to design and implement the remote sensing data processing service component (Remote Sensing User-Defined Function, RS-UDF). RS-UDF enables users to package existing programs, custom functions, and references to existing mature processing technologies. It realizes synchronous and asynchronous calls in the form of interface services transfer. In addition, the framework achieves a remote sensing data scheduling engine (DataboxMR-Engine) based on a two-tier scheduling model, supporting designated node processing tasks, task division and distribution, and fault recovery functions.[Results] Experimental comparison with GeoTrellis, a remote sensing data processing tool based on memory computing, shows that DataboxMR is more efficient and has less system overhead when performing lightweight remote sensing data processing tasks. [Conclusions] DataboxMR is a lightweight and efficient distributed scheduling framework for remote sensing data.

Key words: remote sensing data, distributed scheduling framework, UDF technology, scheduling engine

MENG Xianghai,WANG Xuezhi,ZHAO Jianghua,ZHOU Xiaohua. A Lightweight Remote Sensing Data Distributed Scheduling Framework DataboxMR[J]. Frontiers of Data and Computing, 2021, 3(3): 95-110.

Figures/Tables 15

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Table 1

Table 2

Table 3

Fig.8

Fig.9

Fig.10

Fig.11

Fig.12

References 21

[1]	PHILIP C L, ZHANG C Y. Data-intensive applications, challenges, techniques and technologies: a survey on big data[J]. Information Sciences, 2014, 275:314-347. DOI: 10.1016/j.ins.2014.01.015. doi: 10.1016/j.ins.2014.01.015
[2]	钟运琴, 方金云, 赵晓芳. 大规模时空数据分布式存储方法研究[J]. 高技术通讯, 2013, 23(12):1219-1229.
[3]	CHAOWEI YANG, MICHAEL GOODCHILD, QUN-YING HUANG, et al. Spatial cloud computing: how can the geospatial sciences use and help shape cloud computing?[J]. International Journal of Digital Earth, 2011, 4(4):305-329. doi: 10.1080/17538947.2011.587547
[4]	张福浩, 张明波, 张志然, 等. 分布式空间数据库安全机制探讨[J]. 测绘通报, 2016(1):41-44.
[5]	王文生, 郭雷风. 大数据技术农业应用[J]. 数据与计算发展前沿, 2020, 2(2):101-110.DOI: 10.11871/jfdc.issn.2096-742X.2020.02.008.PID: 21.86101.2/jfdc.2096-742X.2020.02.008.
[6]	张嘉, 白晓飞, 陶超, 张小桐. 大规模空间矢量数据分布式存储与计算优化[J]. 计算机系统应用, 2020, 29(12):251-256. http://www.c-s-a.org.cn/1003-3254/7724.html.
[7]	宋峣, 孙小涓, 胡玉新, 等. 基于流式计算的遥感卫星数据快视处理方法[J]. 计算机工程与应用, 2019, 55(10):77-82.
[8]	刘欢, 陈能成, 陈泽强. 基于Apache Spark的MODIS海表温度反演方法[J]. 计算机系统应用, 2018, 27(9):112-117. http://www.c-s-a.org.cn/1003-3254/6534.html.
[9]	Eldawy A, Mokbel M. SpatialHadoop: A MapReduce framework for spatial data[C]. International Confer-ence on Data Engineering. 2015(5):1352-1363.
[10]	Aji A, Wang F, Vo H, et al. Hadoop-GIS: A High Per-formance Spatial Data Warehousing System over Map-Reduce[J]. Proceedings of the VLDB Endowment, 2013, 6(11):1009-1020. doi: 10.14778/2536222.2536227
[11]	Yu J, Wu J, Sarwat M. GeoSpark: a cluster computing framework for processing large-scale spatial data[C]. The 23rd SIGSPATIAL International Conference. ACM, 2015(11):1-4.
[12]	Tang M, Yu Y, Malluhi Q M, et al. LocationSpark: A distributed in-memory data management system for big spatial data[J]. Proceedings of the VLDB Endowment, 2016, 9(13):1565-1568. doi: 10.14778/3007263.3007310
[13]	Xie D, Li F, Yao B, et al. Simba: spatial in-memory big data analysis[C]. The 24th ACM SIGSPATIAL International Conference. ACM, 2016(10):1-4.
[14]	Wang L, Ma Y, Yan J, et al. PipsCloud: High performance cloud computing for remote sensing big data management and processing[J]. Future Generation Computer Systems, 2016(8):353-368.
[15]	Wulder M A, White J C, Loveland T R, et al. The global Landsat archive: Status, consolidation, and direction[J]. Remote Sensing of Environment, 2016, 185:271-283.DOI: https://doi.org/10.1016/j.rse.2015.11.032. doi: 10.1016/j.rse.2015.11.032
[16]	Apache. GeoTrellis Document[EB/OL].[2020-08-28]. https://geotrellis.readthedocs.io/en/latest/.
[17]	Wiener P.; Simko V.; Nimis J. Taming the Evolution of Big Data and its Technologies in BigGIS—AConceptual Architectural Framework for Spatio-Temporal Analytics at Scale[J]. In Proceedings of the 3rdInternational Confer-ence on Geographical Information Systems Theory, Appli-cations and Management(GISTAM 2017). SCITEPRESS, Porto, Portugal, 27-28 April 2017; pp. 90-101. DOI: 10.5220/0006334200900101.
[18]	Apache. GeoTrellis-Landsat-Tutorial[EB/OL].[2020-08-28]. https://github.com/geotrellis/geotrellis-landsat-tutorial.
[19]	张耀南, 艾鸣浩, 康建芳, 等. 地学大数据处理架构与关键技术研究[J]. 数据与计算发展前沿, 2020, 2(2):91-100.DOI: 10.11871/jfdc.issn.2096-742X.2020.02.007.PID:21.86101.2/jfdc.2096-742X.2020.02.007.
[20]	Wyborn L, Evans B J K. Integrating ‘Big’geoscience data into the petascale national environmental research interoperability platform (NERDIP): Successes and unforeseen challenges[C]. 2015 IEEE International Conference on Big Data (Big Data). IEEE, 2015: 2005-2009. DOI: 10.1109/bigdata.2015.7363981.
[21]	Meng Zhen, Wang Xuezhi, Xie Zhimin, et al. IA: An Inter-active Analysis Service Management Engine in Scien-tific Data Cloud[J]. Frontiers of Data & Computing, 2020, 2(2):31-39.DOI: 10.11871/jfdc.issn.2096-742X.2020.02.003.PID:21.86101.2/jfdc.2096-742X.2020.02.003.

数据名称	数据量
LC08_L1GT_123032_20200819_20200823_01_T2	999MB
LC08_L1TP_121032_20200821_20200905_01_T1	862MB
LC08_L1TP_121032_20200906_20200906_01_RT	950MB
LC08_L1TP_122032_20200812_20200822_01_T1	990MB
LC08_L1TP_122032_20200828_20200905_01_T1	920MB
LC08_L1TP_122033_20200828_20200905_01_T1	871MB
LC08_L1TP_123032_20200803_20200807_01_T1	913MB
LC08_L1TP_123032_20200904_20200904_01_RT	980MB
LC08_L1TP_123033_20200803_20200807_01_T1	980MB
LC08_L1TP_123033_20200904_20200904_01_RT	913MB

框架	读数据(秒)	处理数据(秒)	写数据(秒)
GeoTrellis	276.276	0.309	261.011
DataboxMR	154.133	0.223	12.6814