Hyperspectral Image Classification Method Combining Transformer and Multi-Layer Feature Aggregation

doi:10.11871/jfdc.issn.2096-742X.2023.03.010

Abstract

Abstract:

[Background] In recent years, the convolutional neural network has been widely used in hyper-spectral image classification tasks and achieved excellent classification performance. However, the convolutional neural network still has many limitations. For example, the convolution has a small receiving area and the downsampling operation will cause the loss of spatial information. [Objective] To solve the above problems, this paper proposes a hyperspectral image classification method that combines transformer and multi-layer feature aggregation (TMFANet). [Methods] First, TMFANet uses 3D CNN and 2D CNN layers to extract the shallow spatial-spectral features of the image. Then, a method based on the dense convolution Transformer module is proposed to extract the global features of images. Then, a multi-layer feature aggregation module is proposed to extract image features at different levels. Finally, the extracted abstract features are sent to a classifier for classification. To verify the effectiveness of TMFANet, a series of experiments are carried out on three public data sets, Indian pines, Pavia University and Salinas. [Results] The experimental results show that the classification performance of TMFANet proposed in this paper is better than other current methods.

Key words: Convolution neural network, Image processing, Hyperspectral image classification, Transformer, Characteristic aggr-egation

CHEN Dong, LI Ming, CHEN Shuwen. Hyperspectral Image Classification Method Combining Transformer and Multi-Layer Feature Aggregation[J]. Frontiers of Data and Computing, 2023, 5(3): 138-151, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2023.03.010.

Figures/Tables 13

Fig.1

Fig.2

Table 1

Fig.3

Table 2

Table 3

Table 4

Table 5

Table 6

Fig.4

Fig.5

Fig.6

Table 7

References 19

[1]	E. Pasolli, F. Melgani, D. Tuia, F. Pacifici, and W.J. Emery-SVM Active Learning Approach for Image Classification Using Spatial Information[J]. IEEE Transactions on Geos-cience and Remote Sensing. 2014, 52(4): 2217-2233.
[2]	S. Alim, G. Paolo, A. Jilili, S. Liu, and Z. Miao. Geodesic flow kernel support vector machine for hyperspectral image classification by unsupervised subspace feature transfer[J]. Remote Sensing. 2016, 8(3):234. doi: 10.3390/rs8030234
[3]	P. Du, K. Tan, and X. Xing. Wavelet SVM in Reproduc-ing Kernel Hilbert Space for hyperspectral remote sen-sing image classification[J]. Optics Communications, 2010, 283(24): 4978-4984. doi: 10.1016/j.optcom.2010.08.009
[4]	Y. Chen, H. Jiang, C. Li, X. Jia, and P. Ghamisi. Deep fe-ature extraction and classification of hyperspectral ima-ges based on convolutional Lneural networks[J]. IE-EE Transactions on Geoscience and Remote Sensing, 2016, 54(10): 6232-6251.
[5]	Y. Li, H. Zhang, and Q. Shen. Spectral-spatial class-ification of hyperspectral imagery with 3D convolutional neural network[J]. Remote Sensing, 2017, 9(1):67. doi: 10.3390/rs9010067
[6]	S. K. Roy, G. Krishna, S. R. Dubey, and B. B. Chaudhuri. HybridSN: Exploring 3-D-2-D CNN feature hierarchy for hyperspectral image classification[J]. IEEE Geosc-ience and Remote Sensing Letters, 2020, 17(2): 277-281.
[7]	ZHONG Zilong, LI J, LUO Zhiming, et al. Spectral-spatial residual network for hyperspectral image classi-fication: a 3-D deep learning framework[J]. IEEE tran-sactions on geoscience and remote sensing, 2018, 56(2): 847-858.
[8]	M. R. Haque, and S. Z. Mishu. Spectral-Spatial Feat-ure Extraction Using PCA and Multi-Scale Deep Convo-lutional Neural Network for Hyperspectral Image Class-ification[C]. 2019 22nd International Conference on Com-puter and Information Technology (ICCIT) IEEE, 2020: 468-471.
[9]	M. He, Bo. L, and H. Chen. Multi-scale 3D deep con-volutional neural network for hyperspectral image class-ification[C]. 2017 IEEE International Conference on Image Processing (ICIP) IEEE, 2018:342-346.
[10]	H. Yu, H. Zhang, Y. Liu, K. Zheng, Z. Xu, and C. Xiao. Dual-channel convolution network with image-based global learning framework for hyperspectral image class-ification[J]. IEEE Geoscience and Remote Sensing Lett-ers, 2021, 9:1-5.
[11]	H. Lee and H. Kwon. Going Deeper with Contextual CNN for Hyperspectral Image Classification[J]. IEEE Trans Image Process. 2017, 26(10): 4843-4855. doi: 10.1109/TIP.2017.2725580
[12]	X. Cao, X. Fu, C. Xu, and D. Meng. Deep SpatialSpe-ctral Global Reasoning Network for Hyperspectral Image noising[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 1-14.
[13]	Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C]. Proceedings of the IEEE Conference on Computer Vision And Pattern Recognition, 2018: 7132-7141.
[14]	W. Ma, Q. Yang, Y. Wu, W. Zhao, and X. Zhang. Dou-ble-branch multiattention mechanism network for hyper-spectral image classification[J]. Remote Sens, 2019, 11 (11):1307. doi: 10.3390/rs11111307
[15]	R. Li, S. Zheng, C. Duan, Y. Yang, and X. Wang. Class-ification of hyperspectral image based on double-bra-nch dual-attention mechanism network[J]. Remote Sensing, 2020, 12(3):582. doi: 10.3390/rs12030582
[16]	A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, and N. Houlsby. An image is worth 16x16 words: trans-formers for image recognition at scale[J/OL]. 2020. https://doi.org/10.48550/arXiv.2010.11929.
[17]	D. Hong, Z. Han, J. Yao, L. Gao, B. Zhang, A. Plaza, and J. Chanussot. SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers[J/OL]. IEEE Transactions on Geoscience and Remote Sensing. 2021. https://doi.org/10.1109/TGRS.2021.3130716.
[18]	D. Hong et al. SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers[J]. IEEE Tran-sactions on Geoscience and Remote Sensing, 2022, 60:1-15.
[19]	Z. Zhong, Y. Li, L. Ma, J. Li, and W. S. Zheng. Spectral-Spatial Transformer Network for Hyperspectral Image Classification: A Factorized Architecture Search Frame-work[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:1-15.

标号	Indian Pines		Pavia University		Salinas valley
标号	类别名称	样本数量	类别名称	样本数量	类别名称	样本数量
1	Alfalfa	46	Asphalt	6631	Brocoil-green-weeds_1	2009
2	Corn-notill	1428	Meadows	18649	Brocoil-green-weeds_2	3726
3	Corn-mintill	830	Gravel	2099	Fallow	1976
4	Corn	237	Trees	3064	Fallow-rough-plow	1394
5	Grass/pasture	483	Painted metal sheets	1345	Fallow-smooth	2678
6	Grass/trees	730	Bare Soil	5029	Stubble	3959
7	Grass/pasture-mowed	28	Bitumen	1330	Celery	3579
8	Hay-windrowed	478	Self-Blocking Bricks	3682	Grapes-untrained	11271
9	Oats	20	Shadows	947	Soil-vinyard-develop	6203
10	Soybean-notill	972			Corn-senesced-green-weeds	3278
11	Soybean-mintill	2455			Lettuce-romaine-4wk	1068
12	Soybean-clean	593			Lettuce-romaine-5wk	1927
13	Wheat	205			Lettuce-romaine-6wk	916
14	Woods	1265			Lettuce-romaine-7wk	1070
15	Bldg-Grass-Tree-Drivers	386			Vinyard-untrained	7268
16	Stone-Steel-Towers	93			Vinyard-vertical-trellis	1807
总数		10249		42776		54129

网络	计算	感受野
Conv3D	$l_{0}+(7-1)\times 1$	7
Conv2D	$l_{1}+(7-1)\times 1$	13
Transformer	$l_{2}+(81-1)\times 1$	93
1×1Conv	$l_{3}+(1-1)\times 1$	93
3×3Conv	$l_{4}+(3-1)\times 1$	95
5×5Conv	$l_{5}+(5-1)\times 1$	99
7×7Conv	$l_{6}+(7-1)\times 1$	105

Dropout Rate	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
Indian Pines	96.15	96.59	96.79	95.49	94.85	96.25	96.16	95.76	96.00
Pavia University	96.16	96.65	95.84	96.05	96.38	94.78	95.46	96.18	95.79
Salinas	97.08	96.88	97.12	96.66	97.80	97.51	97.02	97.43	96.81

类别	SVM^[1]	CDCNN^[11]	2DCNN^[4]	3DCNN^[5]	SSRN^[7]	HybridSN^[6]	ViT^[16]	SF^[18]	SSTN^[19]	TMFANet
1	25.92	66.67	100	100	100	52.89	25.00	30.35	93.25	97.29
2	62.35	79.47	61.76	72.90	94.05	80.37	58.62	69.15	98.10	97.47
3	69.33	58.44	57.83	58.15	95.66	83.32	40.30	64.39	91.60	96.70
4	59.13	72.04	82.75	93.75	96.03	77.43	43.95	56.12	96.16	99.48
5	86.00	94.93	92.16	94.25	96.61	89.02	59.66	80.07	99.59	98.83
6	84.52	94.91	69.10	74.17	96.71	96.72	84.31	88.37	99.30	97.60
7	86.61	54.83	100	100	90.90	25.28	60.00	25.00	97.90	86.95
8	87.55	91.18	93.72	87.42	97.11	93.40	78.51	84.06	97.24	95.84
9	53.84	32.25	100	100	85.71	33.33	50.00	69.44	99.46	98.61
10	72.80	69.69	92.24	71.16	91.91	77.44	41.58	71.78	93.97	95.54
11	71.44	81.67	92.29	67.94	96.77	85.89	63.12	72.04	99.63	97.87
12	64.92	57.30	75.64	66.76	95.26	84.00	45.27	61.00	100	96.08
13	90.35	97.06	100	99.41	100	94.39	66.27	89.13	100	99.45
14	90.54	85.03	87.72	89.00	96.10	93.10	83.07	90.93	96.14	94.27
15	70.29	96.52	91.11	80.53	94.29	85.64	64.30	66.76	79.98	97.07
16	98.41	86.73	100	100	94.38	77.67	64.00	97.07	100	94.31
OA	74.74	79.31	70.54	74.75	95.57	85.70	61.81	71.64	94.70	96.79
AA	73.24	76.11	83.52	84.74	95.09	76.87	58.00	66.37	96.39	95.57
Kappa	70.97	76.36	65.65	70.75	94.95	83.65	56.25	67.28	94.11	96.34

类别	SVM^[1]	CDCNN^[11]	2DCNN^[4]	3DCNN^[5]	SSRN^[7]	HybridSN^[6]	ViT^[16]	SF^[18]	SSTN^[19]	TMFANet
1	80.26	88.03	83.60	81.69	93.62	88.08	78.61	76.87	68.76	97.32
2	86.94	95.99	92.28	91.97	97.87	95.61	88.82	82.76	89.54	98.52
3	71.73	77.93	85.90	68.96	99.17	75.89	56.75	34.90	15.45	89.32
4	96.44	84.20	98.64	97.81	99.25	96.03	96.97	84.62	99.49	99.54
5	90.85	93.37	94.40	97.40	99.69	98.74	86.49	95.20	97.56	97.22
6	77.02	95.53	89.36	84.44	96.99	93.79	87.77	79.53	89.39	96.02
7	69.70	73.52	93.72	90.42	93.49	80.97	60.12	26.12	11.44	98.88
8	67.30	79.19	75.40	78.89	83.55	80.00	73.12	63.95	66.05	87.18
9	99.89	71.00	99.86	99.87	98.79	90.87	97.84	99.24	57.48	100
OA	83.07	90.28	89.28	88.14	95.84	91.31	84.24	79.41	81.59	96.65
AA	82.24	84.31	90.35	87.94	95.82	88.89	80.72	71.47	66.13	96.00
Kappa	77.07	87.04	85.54	84.02	94.46	88.41	78.62	71.76	74.87	95.55