Multispectral Remote Sensing Image Pansharpening Method Based on Shallow-Deep Convolutional Recurrent Neural Network

doi:10.11871/jfdc.issn.2096-742X.2025.05.011

Abstract

Abstract:

[Objective] Multispectral image panchromatic sharpening refers to the fusion of high spatial resolution panchromatic images (PAN) and low spatial resolution multispectral images (MS) to obtain hyperspectral and spatially resolution multispectral images. Many existing panchromatic sharpening methods based on deep learning tend to ignore the local dependence and global correlation among various bands of panchromatic images and multispectral images. [Methods] In response to the above problems, this paper proposes a multispectral image pansharpening method based on a shallow-deep convolutional recurrent neural network, which consists of a shallow feature extraction sub-network and a deep feature fusion sub-network. By using recurrent neural networks to simulate the interaction between bands, the fusion effect is effectively enhanced. [Results] Tests were conducted on multiple datasets in the resolution reduction and full resolution experiments. The experimental results show that the proposed method is superior to the traditional full-color sharpening method in the quality of the fusion results. [Conclusion] The shallow feature extraction subnetwork extracts shallow features from PAN and MS images. The deep feature fusion subnetwork captures local and global correlations by establishing intra-view and inter-view relationships between bands, improving the performance of panchromatic sharpening of multispectral images.

Key words: pansharpening technology, deep learning, convolutional recurrent neural network, satellite remote sensing image

WANG Peng,YANG Xiaofeng,HE Zhongchen,DU Jun. Multispectral Remote Sensing Image Pansharpening Method Based on Shallow-Deep Convolutional Recurrent Neural Network[J]. Frontiers of Data and Computing, 2025, 7(5): 138-152, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2025.05.011.

Figures/Tables 26

Fig.1

Fig.2

Table 1

Table 2

Fig.3

Fig.4

Fig.5

Table 3

Table 4

Table 5

Fig.6

Fig.7

Fig.8

Fig.9

Fig.10

Fig.11

Fig.12

Fig.13

Table 6

Table 7

Table 8

Table 9

Table 10

Table 11

Fig.14

Table 12

References 19

[1]	SCHMITT M, ZHU X X. Data fusion and remote sensing: an ever-growing relationship[J]. IEEE Geoscience and Remote Sensing Magazine, 2016, 4(4): 6-23.
[2]	张良培, 沈焕锋. 遥感数据融合的进展与前瞻[J]. 遥感学报, 2016, 20(5): 1050-1061.
[3]	ESLAMI M, MOHAMMADZADEH A. Developing a spectral-based strategy for urban object detection from airborne hyperspectral TIR and visible data[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2016, 9(5): 1808-1816.
[4]	敬忠良, 肖刚, 李振华. 图像融合-理论与应用[M]. 北京: 高等教育出版社, 2007.
[5]	CARPER W, LILLESAND T, KIEFER R. The use of intensity-hue-saturation transformations for merging SPOT panchromatic and multispectral image data[J]. Photogrammetric Engineering and Remote Sensing, 1990, 56(4): 459-467.
[6]	CHAVEZ P J R, KWARTENG A. Extracting spectral contrast in Landsat Thematic Mapper image data using selective principal component analysis[J]. Photogrammetric Engineering and Remote Sensing, 1989, 55(3): 339-348.
[7]	SHETTIGARA V. A generalized component substitution technique for spatial enhancement of multispectral images using a higher resolution data set[J]. Photogrammetric Engineering and Remote Sensing, 1992, 58(5): 561-567.
[8]	GILLESPIE A R, KAHLE A B, WALKER R E. Ⅱ. Color enhancement of highly correlated images. Channel ratio and “chromaticity” transformation techniques-ScienceDirect[J]. Remote Sensing of Environment, 1987, 22(3): 343-365.
[9]	VIVONE G, RESTAINO R, MURA M D, et al. Contrast and error-based fusion schemes for multispecral image pansharpening[J]. IEEE Geoscience and Remote Sensing Letters, 2014, 11(5): 930-934.
[10]	WALD L, RANCHIN T, MANGOLINI M. Fusion of satellite images of different spatial resolutions: assessing the quality of resulting images[J]. Photogrammetric Engineering and Remote Sensing, 1997, 63(6):691-699.
[11]	MALLA S G. A theory for multiresolution signal decomposition: the wavelet representation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1989, 11(7): 674-693.
[12]	DO M N, VETTERLI M. The contourlet transform: an efficient directional multiresolution image representation[J]. IEEE Transactions on image Processing, 2005, 14(12): 2091-2106. doi: 10.1109/tip.2005.859376 pmid: 16370462
[13]	BALLESTER C, CASELLES V, IGUAL L, et al. A variational model for P+XS image fusion[J]. Intemational Journal of Computer Vision, 2006, 69(1):43-58.
[14]	MASI G, COZZOLINO D, VERDOLIVA L, et al. Pansharpening by convolutional neural networks[J]. Remote Sensing, 2016, 8(7) 594: 1-594: 22.
[15]	LIU Q, ZHOU H, XU Q, et al. Psgan: A generative adversarial network for remote sensing image pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(12): 10227-10242.
[16]	HU J, HU P, KANG X, et al. FAN. Pan-Sharpening via Multiscale Dynamic Convolutional Neural Network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(3): 2231-2244.
[17]	H. ZHOU, Q. LIU AND Y. WANG. PGMAN: An Unsupervised Generative Multiadversarial Network for Pansharpening[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14: 6316-6327.
[18]	ZHAO Z, ZHANG R. A small sample bearing fault diagnosis method based on ConvGRU relation network[J]. Measurement Science and Technology, 2024, 35(6).
[19]	MENG X, XIONG Y M, SHAO F, et al. A Large-Scale Benchmark Data Set for Evaluating Pansharpening Performance: Overview and Implementation[J]. IEEE Geoscience and Remote Sensing Magazine, 2021, 9(1): 18-52.

传感器		空间分辨率	光谱分辨率	尺寸	场景内容
QuickBird	PAN	0.6m	1 波段	1,024×1,024	建筑，植被，水体，土地
QuickBird	MS	2.4m	4 波段	256×256	建筑，植被，水体，土地
WorldView-4	PAN	0.31m	1 波段	1,024×1,024	建筑，森林，水体
WorldView-4	MS	1.24m	4 波段	256×256	建筑，森林，水体
WorldView-2	PAN	0.5m	1 波段	1,024×1,024	建筑，水体，植被，沙漠
WorldView-2	MS	2m	8 波段	256×256	建筑，水体，植被，沙漠

方法	PNN	PanNet	MSDCNN	TFNet	GPPNN	MCRNN
迭代次数	1.12×10⁶	2.5×10⁵	2.5×10⁵	1.1×10⁶	2.4×10⁵	1.6×10⁵
批大小	128	16	64	32	16	64
核尺寸	9×9 5×5	3×3	7×7 3×3 1×1	3×3 2×2 1×1	3×3	3×3 1×1
学习率	0.00001	0.001	0.1	0.0001	0.00005	0.00001
优化方法	SGD	SGD	SGD	Adam	Adam	Adam
损失函数	MSE Loss	MSE Loss	MSE Loss	L1 Loss	L1 Loss	MSE Loss

方法	Quick Bird
方法	SAM	ERGAS	Q4
Reference	0.00	0.00	1.00
EXP	1.18±0.30	1.18±0.36	0.72±0.04
Brovey	0.93±0.19	1.17±0.31	0.75±0.07
CNMF	0.93±0.2	1.13±0.33	0.79±0.05
GS	0.79±0.13	0.97±0.18	0.80±0.07
GSA	0.69±0.14	0.88±0.18	0.82±0.08
SFIM	0.79±0.15	0.92±0.20	0.81±0.07
PRACS	1.08±0.26	1.25±0.36	0.76±0.05
PNN	0.73±0.12	0.80±0.15	0.84±0.07
PanNet	0.70±0.12	0.86±0.18	0.81±0.09
MSDCNN	0.68±0.14	0.76±0.19	0.84±0.07
TFNet	0.64±0.12	0.83±0.16	0.83±0.07
GPPNN	0.55±0.16	0.84±0.25	0.84±0.10
MCRNN	0.49±0.12	0.72±0.16	0.85±0.07

方法	WorldView-4
方法	SAM	ERGAS	Q4
Reference	0.00	0.00	1.00
EXP	3.61±0.74	2.26±0.82	0.70±0.09
Brovey	3.39±0.55	2.30±0.72	0.75±0.14
CNMF	3.07±0.51	2.15±0.67	0.80±0.14
GS	3.56±0.88	2.88±1.12	0.75±0.16
GSA	3.11±0.63	2.26±0.64	0.80±0.13
SFIM	3.64±0.90	2.41±0.79	0.81±0.12
PRACS	2.83±0.70	2.80±1.04	0.78±0.10
PNN	2.33±0.62	2.45±0.81	0.74±0.15
PanNet	2.29±0.55	2.42±0.49	0.71±0.18
MSDCNN	2.06±0.58	2.03±0.64	0.77±0.13
TFNet	2.24±0.61	2.38±0.89	0.74±0.10
GPPNN	2.07±0.54	2.08±0.57	0.78±0.13
MCRNN	1.68±0.42	1.82±0.58	0.83±0.11

方法	WorldView-2
方法	SAM	ERGAS	Q8
Reference	0.00	0.00	1.00
EXP	5.34±0.58	6.62±1.17	0.58±0.05
Brovey	5.31±0.63	6.56±1.05	0.72±0.07
CNMF	4.03±0.52	5.71±0.93	0.73±0.08
GS	5.38±0.64	7.01±1.02	0.71±0.06
GSA	4.61±0.55	6.90±1.37	0.78±0.07
SFIM	6.78±3.88	6.50±1.21	0.81±0.08
PRACS	5.21±0.61	7.52±1.34	0.78±0.06
PNN	3.52±0.37	5.83±0.86	0.73±0.07
PanNet	3.09±0.34	4.93±0.75	0.81±0.09
MSDCNN	2.96±0.31	4.76±0.59	0.81±0.09
TFNet	2.99±0.23	4.88±0.62	0.81±0.05
GPPNN	2.96±0.31	4.76±0.59	0.81±0.09
MCRNN	2.92±0.32	4.71±0.66	0.86±0.09