结合Transformer和多层特征聚合的高光谱图像分类算法

doi:10.11871/jfdc.issn.2096-742X.2023.03.010

数据与计算发展前沿 ›› 2023, Vol. 5 ›› Issue (3): 138-151.

CSTR: 32002.14.jfdc.CN10-1649/TP.2023.03.010

doi: 10.11871/jfdc.issn.2096-742X.2023.03.010

结合Transformer和多层特征聚合的高光谱图像分类算法

陈栋¹(),李明^1,^*(),陈淑文²

1.重庆师范大学，计算机与信息科学学院，重庆 401331
2.重庆师范大学，数学科学学院，重庆 401331

收稿日期:2022-10-10 出版日期:2023-06-20 发布日期:2023-06-21
通讯作者: *李明（E-mail: 20131052@cqnu.edu.com)
作者简介:陈栋，重庆师范大学计算机与信息科学学院，硕士研究生，主要研究方向为深度学习、计算机视觉。
本文中主要承担的任务是算法设计、实现与论文撰写。
Chen Dong is a master’s student at the School of Computer and Information Science, Chongqing Normal University. His research interests include deep learning and computer vision.
In this paper, he is mainly responsible for algorithm design, implementation, and paper writing.
E-mail: 1593056183@qq.com|李明，重庆师范大学计算机与信息科学学院，博士，教授，硕士生导师，教育部高等学校电子商务类专业教学指导委员会委员，重庆市计算机学会常务理事，电子商务国家一流专业带头人。主要研究方向为计算机视觉、大数据与电子商务。
本文中主要承担的任务是研究指导。
Li Ming, Ph.D., is a full professor and master tutor at the School of Computer and Information Science, Chongqing Nor-mal University. He is also a member of the Teaching Steer-ing Committee of E-Commerce Specialty in Colleges and Univer-sities of the Ministry of Education. His main research interests include computer vision, big data, and e-commerce.
In this paper, he is mainly responsible for research guidance.
E-mail: 20131052@cqnu.edu.com
基金资助:
国家自然科学基金(61877051);国家自然科学基金(61170192);重庆市科委重点项目(cstc2017zdcy-zdyf0366);重庆市教委项目(113143);重庆市研究生教改重点项目(yjg182022)

Hyperspectral Image Classification Method Combining Transformer and Multi-Layer Feature Aggregation

CHEN Dong¹(),LI Ming^1,^*(),CHEN Shuwen²

1. College of Computer and Information Science, Chongqing Normal University, Chongqing 401331, China
2. College of Mathematical Sciences, Chongqing Normal University, Chongqing 401331, China

Received:2022-10-10 Online:2023-06-20 Published:2023-06-21

摘要/Abstract

摘要：

【应用背景】 近年来，卷积神经网络被广泛应用在高光谱图像分类任务中，并获得了优异的分类性能。然而，卷积神经网络依然存在着较多的局限性。例如，卷积接受域较小、降采样操作会带来空间信息丢失等。【目的】 为了解决上述问题，本文提出了一种结合Transformer和多层特征聚合的高光谱图像分类方法（Transformer and Multi-Layer Feature Aggregation Network, TMFANet）。【方法】 首先，TMFANet采用二维卷积（2DConv）和三维卷积（3DConv）层提取图像的浅层空间光谱特征。接着，提出了一种基于密集卷积Transformer模块来提取图像的全局特征。然后，提出了一种多层特征聚合模块来提取不同层次的图像特征，最后将提取的抽象特征输送到分类器中进行分类。为了验证TMFANet的有效性，在Indian Pines、Pavia University和Salinas三个公开的数据集上进行了一系列实验。【结果】 实验结果表明，本文提出TMFANet的分类性能比其他先进的方法具有更为优异的性能和泛化能力。

关键词: 卷积神经网络, 图像处理, 高光谱图像分类, Transformer, 特征聚合

Abstract:

[Background] In recent years, the convolutional neural network has been widely used in hyper-spectral image classification tasks and achieved excellent classification performance. However, the convolutional neural network still has many limitations. For example, the convolution has a small receiving area and the downsampling operation will cause the loss of spatial information. [Objective] To solve the above problems, this paper proposes a hyperspectral image classification method that combines transformer and multi-layer feature aggregation (TMFANet). [Methods] First, TMFANet uses 3D CNN and 2D CNN layers to extract the shallow spatial-spectral features of the image. Then, a method based on the dense convolution Transformer module is proposed to extract the global features of images. Then, a multi-layer feature aggregation module is proposed to extract image features at different levels. Finally, the extracted abstract features are sent to a classifier for classification. To verify the effectiveness of TMFANet, a series of experiments are carried out on three public data sets, Indian pines, Pavia University and Salinas. [Results] The experimental results show that the classification performance of TMFANet proposed in this paper is better than other current methods.

Key words: Convolution neural network, Image processing, Hyperspectral image classification, Transformer, Characteristic aggr-egation

陈栋, 李明, 陈淑文. 结合Transformer和多层特征聚合的高光谱图像分类算法[J]. 数据与计算发展前沿, 2023, 5(3): 138-151.

CHEN Dong, LI Ming, CHEN Shuwen. Hyperspectral Image Classification Method Combining Transformer and Multi-Layer Feature Aggregation[J]. Frontiers of Data and Computing, 2023, 5(3): 138-151, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2023.03.010.

图/表 13

图1

图2

表1

图3

表2

表3

表4

表5

表6

图4

图5

图6

表7

参考文献 19

[1]	E. Pasolli, F. Melgani, D. Tuia, F. Pacifici, and W.J. Emery-SVM Active Learning Approach for Image Classification Using Spatial Information[J]. IEEE Transactions on Geos-cience and Remote Sensing. 2014, 52(4): 2217-2233.
[2]	S. Alim, G. Paolo, A. Jilili, S. Liu, and Z. Miao. Geodesic flow kernel support vector machine for hyperspectral image classification by unsupervised subspace feature transfer[J]. Remote Sensing. 2016, 8(3):234. doi: 10.3390/rs8030234
[3]	P. Du, K. Tan, and X. Xing. Wavelet SVM in Reproduc-ing Kernel Hilbert Space for hyperspectral remote sen-sing image classification[J]. Optics Communications, 2010, 283(24): 4978-4984. doi: 10.1016/j.optcom.2010.08.009
[4]	Y. Chen, H. Jiang, C. Li, X. Jia, and P. Ghamisi. Deep fe-ature extraction and classification of hyperspectral ima-ges based on convolutional Lneural networks[J]. IE-EE Transactions on Geoscience and Remote Sensing, 2016, 54(10): 6232-6251.
[5]	Y. Li, H. Zhang, and Q. Shen. Spectral-spatial class-ification of hyperspectral imagery with 3D convolutional neural network[J]. Remote Sensing, 2017, 9(1):67. doi: 10.3390/rs9010067
[6]	S. K. Roy, G. Krishna, S. R. Dubey, and B. B. Chaudhuri. HybridSN: Exploring 3-D-2-D CNN feature hierarchy for hyperspectral image classification[J]. IEEE Geosc-ience and Remote Sensing Letters, 2020, 17(2): 277-281.
[7]	ZHONG Zilong, LI J, LUO Zhiming, et al. Spectral-spatial residual network for hyperspectral image classi-fication: a 3-D deep learning framework[J]. IEEE tran-sactions on geoscience and remote sensing, 2018, 56(2): 847-858.
[8]	M. R. Haque, and S. Z. Mishu. Spectral-Spatial Feat-ure Extraction Using PCA and Multi-Scale Deep Convo-lutional Neural Network for Hyperspectral Image Class-ification[C]. 2019 22nd International Conference on Com-puter and Information Technology (ICCIT) IEEE, 2020: 468-471.
[9]	M. He, Bo. L, and H. Chen. Multi-scale 3D deep con-volutional neural network for hyperspectral image class-ification[C]. 2017 IEEE International Conference on Image Processing (ICIP) IEEE, 2018:342-346.
[10]	H. Yu, H. Zhang, Y. Liu, K. Zheng, Z. Xu, and C. Xiao. Dual-channel convolution network with image-based global learning framework for hyperspectral image class-ification[J]. IEEE Geoscience and Remote Sensing Lett-ers, 2021, 9:1-5.
[11]	H. Lee and H. Kwon. Going Deeper with Contextual CNN for Hyperspectral Image Classification[J]. IEEE Trans Image Process. 2017, 26(10): 4843-4855. doi: 10.1109/TIP.2017.2725580
[12]	X. Cao, X. Fu, C. Xu, and D. Meng. Deep SpatialSpe-ctral Global Reasoning Network for Hyperspectral Image noising[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 1-14.
[13]	Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C]. Proceedings of the IEEE Conference on Computer Vision And Pattern Recognition, 2018: 7132-7141.
[14]	W. Ma, Q. Yang, Y. Wu, W. Zhao, and X. Zhang. Dou-ble-branch multiattention mechanism network for hyper-spectral image classification[J]. Remote Sens, 2019, 11 (11):1307. doi: 10.3390/rs11111307
[15]	R. Li, S. Zheng, C. Duan, Y. Yang, and X. Wang. Class-ification of hyperspectral image based on double-bra-nch dual-attention mechanism network[J]. Remote Sensing, 2020, 12(3):582. doi: 10.3390/rs12030582
[16]	A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, and N. Houlsby. An image is worth 16x16 words: trans-formers for image recognition at scale[J/OL]. 2020. https://doi.org/10.48550/arXiv.2010.11929.
[17]	D. Hong, Z. Han, J. Yao, L. Gao, B. Zhang, A. Plaza, and J. Chanussot. SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers[J/OL]. IEEE Transactions on Geoscience and Remote Sensing. 2021. https://doi.org/10.1109/TGRS.2021.3130716.
[18]	D. Hong et al. SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers[J]. IEEE Tran-sactions on Geoscience and Remote Sensing, 2022, 60:1-15.
[19]	Z. Zhong, Y. Li, L. Ma, J. Li, and W. S. Zheng. Spectral-Spatial Transformer Network for Hyperspectral Image Classification: A Factorized Architecture Search Frame-work[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:1-15.

标号	Indian Pines		Pavia University		Salinas valley
标号	类别名称	样本数量	类别名称	样本数量	类别名称	样本数量
1	Alfalfa	46	Asphalt	6631	Brocoil-green-weeds_1	2009
2	Corn-notill	1428	Meadows	18649	Brocoil-green-weeds_2	3726
3	Corn-mintill	830	Gravel	2099	Fallow	1976
4	Corn	237	Trees	3064	Fallow-rough-plow	1394
5	Grass/pasture	483	Painted metal sheets	1345	Fallow-smooth	2678
6	Grass/trees	730	Bare Soil	5029	Stubble	3959
7	Grass/pasture-mowed	28	Bitumen	1330	Celery	3579
8	Hay-windrowed	478	Self-Blocking Bricks	3682	Grapes-untrained	11271
9	Oats	20	Shadows	947	Soil-vinyard-develop	6203
10	Soybean-notill	972			Corn-senesced-green-weeds	3278
11	Soybean-mintill	2455			Lettuce-romaine-4wk	1068
12	Soybean-clean	593			Lettuce-romaine-5wk	1927
13	Wheat	205			Lettuce-romaine-6wk	916
14	Woods	1265			Lettuce-romaine-7wk	1070
15	Bldg-Grass-Tree-Drivers	386			Vinyard-untrained	7268
16	Stone-Steel-Towers	93			Vinyard-vertical-trellis	1807
总数		10249		42776		54129

网络	计算	感受野
Conv3D	$l_{0}+(7-1)\times 1$	7
Conv2D	$l_{1}+(7-1)\times 1$	13
Transformer	$l_{2}+(81-1)\times 1$	93
1×1Conv	$l_{3}+(1-1)\times 1$	93
3×3Conv	$l_{4}+(3-1)\times 1$	95
5×5Conv	$l_{5}+(5-1)\times 1$	99
7×7Conv	$l_{6}+(7-1)\times 1$	105

Dropout Rate	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
Indian Pines	96.15	96.59	96.79	95.49	94.85	96.25	96.16	95.76	96.00
Pavia University	96.16	96.65	95.84	96.05	96.38	94.78	95.46	96.18	95.79
Salinas	97.08	96.88	97.12	96.66	97.80	97.51	97.02	97.43	96.81

类别	SVM^[1]	CDCNN^[11]	2DCNN^[4]	3DCNN^[5]	SSRN^[7]	HybridSN^[6]	ViT^[16]	SF^[18]	SSTN^[19]	TMFANet
1	25.92	66.67	100	100	100	52.89	25.00	30.35	93.25	97.29
2	62.35	79.47	61.76	72.90	94.05	80.37	58.62	69.15	98.10	97.47
3	69.33	58.44	57.83	58.15	95.66	83.32	40.30	64.39	91.60	96.70
4	59.13	72.04	82.75	93.75	96.03	77.43	43.95	56.12	96.16	99.48
5	86.00	94.93	92.16	94.25	96.61	89.02	59.66	80.07	99.59	98.83
6	84.52	94.91	69.10	74.17	96.71	96.72	84.31	88.37	99.30	97.60
7	86.61	54.83	100	100	90.90	25.28	60.00	25.00	97.90	86.95
8	87.55	91.18	93.72	87.42	97.11	93.40	78.51	84.06	97.24	95.84
9	53.84	32.25	100	100	85.71	33.33	50.00	69.44	99.46	98.61
10	72.80	69.69	92.24	71.16	91.91	77.44	41.58	71.78	93.97	95.54
11	71.44	81.67	92.29	67.94	96.77	85.89	63.12	72.04	99.63	97.87
12	64.92	57.30	75.64	66.76	95.26	84.00	45.27	61.00	100	96.08
13	90.35	97.06	100	99.41	100	94.39	66.27	89.13	100	99.45
14	90.54	85.03	87.72	89.00	96.10	93.10	83.07	90.93	96.14	94.27
15	70.29	96.52	91.11	80.53	94.29	85.64	64.30	66.76	79.98	97.07
16	98.41	86.73	100	100	94.38	77.67	64.00	97.07	100	94.31
OA	74.74	79.31	70.54	74.75	95.57	85.70	61.81	71.64	94.70	96.79
AA	73.24	76.11	83.52	84.74	95.09	76.87	58.00	66.37	96.39	95.57
Kappa	70.97	76.36	65.65	70.75	94.95	83.65	56.25	67.28	94.11	96.34

类别	SVM^[1]	CDCNN^[11]	2DCNN^[4]	3DCNN^[5]	SSRN^[7]	HybridSN^[6]	ViT^[16]	SF^[18]	SSTN^[19]	TMFANet
1	80.26	88.03	83.60	81.69	93.62	88.08	78.61	76.87	68.76	97.32
2	86.94	95.99	92.28	91.97	97.87	95.61	88.82	82.76	89.54	98.52
3	71.73	77.93	85.90	68.96	99.17	75.89	56.75	34.90	15.45	89.32
4	96.44	84.20	98.64	97.81	99.25	96.03	96.97	84.62	99.49	99.54
5	90.85	93.37	94.40	97.40	99.69	98.74	86.49	95.20	97.56	97.22
6	77.02	95.53	89.36	84.44	96.99	93.79	87.77	79.53	89.39	96.02
7	69.70	73.52	93.72	90.42	93.49	80.97	60.12	26.12	11.44	98.88
8	67.30	79.19	75.40	78.89	83.55	80.00	73.12	63.95	66.05	87.18
9	99.89	71.00	99.86	99.87	98.79	90.87	97.84	99.24	57.48	100
OA	83.07	90.28	89.28	88.14	95.84	91.31	84.24	79.41	81.59	96.65
AA	82.24	84.31	90.35	87.94	95.82	88.89	80.72	71.47	66.13	96.00
Kappa	77.07	87.04	85.54	84.02	94.46	88.41	78.62	71.76	74.87	95.55

结合Transformer和多层特征聚合的高光谱图像分类算法

Hyperspectral Image Classification Method Combining Transformer and Multi-Layer Feature Aggregation

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 13

参考文献 19

相关文章 8

编辑推荐

Metrics

本文评价

类别	SVM^[1]	CDCNN^[11]	2DCNN^[4]	3DCNN^[5]	SSRN^[7]	HybridSN^[6]	ViT^[16]	SF^[18]	SSTN^[19]	TMFANet
1	99.74	89.70	98.87	99.32	99.89	99.94	97.93	97.23	99.87	100
2	99.64	85.97	96.75	99.48	100	99.32	98.16	98.25	90.98	100
3	92.47	92.67	97.34	95.97	98.86	99.03	94.50	92.24	98.88	98.77
4	97.76	96.34	97.26	97.90	95.77	98.70	94.82	96.23	99.16	98.90
5	95.40	95.12	99.01	99.29	98.81	97.03	93.43	96.34	86.99	97.72
6	100	96.25	95.99	99.97	100	99.57	99.36	99.50	98.52	99.94
7	97.59	86.94	98.36	97.88	100	99.68	98.32	96.74	92.83	99.71
8	71.67	90.22	74.07	83.23	92.44	92.87	84.11	74.07	68.14	97.91
9	98.76	94.81	99.39	99.90	99.73	99.43	97.75	97.86	90.81	99.81
10	89.50	93.14	92.70	90.07	98.09	97.57	83.54	84.97	80.86	98.88
11	94.21	74.89	97.66	96.17	100	97.49	83.19	89.78	97.77	99.80
12	96.27	99.13	97.56	95.97	99.84	99.28	92.27	94.88	97.16	99.78
13	96.81	99.55	96.82	96.30	100	98.89	96.92	96.29	100	100
14	94.79	99.51	97.70	97.07	100	94.68	95.04	95.07	80.57	91.62
15	68.35	68.91	80.05	79.76	97.83	90.90	79.75	72.83	78.87	90.95
16	99.87	100	98.98	99.56	100	99.35	96.59	96.87	100	100
OA	87.89	87.97	89.91	92.36	97.69	96.54	90.89	88.23	84.64	97.80
AA	93.30	91.45	94.91	95.49	98.83	97.73	92.85	92.45	91.34	98.37
Kappa	86.47	86.64	88.71	91.48	97.42	96.15	89.86	86.87	82.76	97.56

方法	空间尺寸	参数	运算
CDCNN^[11]	9×9	4.24M	5.81M
2DCNN^[4]	9×9	0.036M	0.11M
3DCNN^[5]	9×9	0.77M	93.82M
SSRN^[7]	9×9	1.39M	23.40M
HybridSN^[6]	9×9	1.54M	1.90M
ViT^[16]	9×9	0.92M	1.28M
TMFANet	9×9	1.78M	10.62M

[1]	石雪梅,朱克亮,张祥民,张树涛,陈良锋. 基于生成对抗网络的有遮挡人脸修复方法[J]. 数据与计算发展前沿, 2022, 4(4): 123-131.
[2]	刘琦玮,李俊,顾蓓蓓,赵泽方. TSAIE：图像增强文本的多模态情感分析模型[J]. 数据与计算发展前沿, 2022, 4(3): 131-140.
[3]	张猛,李健. 鸟类音频数据预处理方法[J]. 数据与计算发展前沿, 2021, 3(5): 130-140.
[4]	肖楠,周明珠,邢军,罗泽,李晓辉. 基于高分辨率网络和注意力机制的真伪卷烟包装鉴别[J]. 数据与计算发展前沿, 2021, 3(5): 118-129.
[5]	石晓霖,毕重科,黄元琪,邓亮,王岳青,王昉. 基于虚拟视点的大规模数值模拟的可交互式原位可视化[J]. 数据与计算发展前沿, 2021, 3(4): 30-43.
[6]	陈子健,李俊,岳兆娟,赵泽方. 基于自编码器与属性信息的混合推荐模型[J]. 数据与计算发展前沿, 2021, 3(3): 148-155.
[7]	刘晓东,倪浩然. 深度学习技术在学科融合研究中的应用[J]. 数据与计算发展前沿, 2020, 2(5): 99-109.
[8]	欧阳与点,谢鲲. 网络性能数据恢复算法[J]. 数据与计算发展前沿, 2020, 2(3): 55-65.