Video Action Recognition Model Based on Attention and Relative Average Discriminator

doi:10.11871/jfdc.issn.2096-742X.2025.02.011

Abstract

Abstract:

[Objective] In the task of video action recognition, how to obtain more action features is a key research issue. Video action features include temporal and spatial features. If the model cannot extract enough spatiotemporal features, it will seriously affect the results of video action recognition. [Methods] Super-resolution technology mainly serves to convert low-resolution images into high-resolution images; some researchers have applied super-resolution technology to the field of low-resolution video action recognition. However, these methods mainly focus on enhancing the resolution of video frame images from the overall perspective of the video, paying insufficient attention to the features of the action itself. To address this issue, this paper proposes a super-resolution network model based on attention mechanisms and relative average discriminators (ARADSP). [Results] The proposed method focuses on action features in videos with more spatiotemporal information through attention mechanisms and improves data quality and stability through relative average discriminators. Finally, experiments are conducted on HMDB51, UCF101, and Something-Something V1&V2 datasets. Extensive experimental results demonstrate the effectiveness of the proposed method in video action recognition.

Key words: attention, supper-resolution, generative adversarial network, realistic average discriminator, low-resolution action recognition

WANG Qijun,LIU Tinglong. Video Action Recognition Model Based on Attention and Relative Average Discriminator[J]. Frontiers of Data and Computing, 2025, 7(2): 109-119, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2025.02.011.

Figures/Tables 9

Fig.1

Fig.2

Table 1

Fig.3

Fig.4

Fig.5

Table 2

Table 3

Table 4

References 39

[1]	吴建超, 王利民, 武港山. 视频群体行为识别综述[J]. 软件学报, 2023, 34(2): 964-984.
[2]	丁静, 舒祥波, 黄捧, 等. 基于多模态多粒度图卷积网络的老年人日常行为识别[J]. 软件学报, 2023, 34(5): 2350-64.
[3]	唐超, 王文剑, 李伟, 等. 基于多学习器协同训练模型的人体行为识别方法[J]. 软件学报, 2015, 26(11): 2939-50.
[4]	ATAER-CANSIZOGLU E, JONES M, ZHANG Z, et al. Verification of very low-resolution faces using an identity-preserving deep face super-resolution network[J]. arXiv preprint arXiv:190310974, 2019.
[5]	BAI Y, ZHANG Y, DING M, et al. Sod-mtgan: Small object detection via multi-task generative adversarial network[C]// Proceedings of the Proceedings of the European Conference on Computer Vision (ECCV), F, 2018.
[6]	WANG Z, YE M, YANG F, et al. Cascaded SR-GAN for scale-adaptive low resolution person re-identification[C]// Proceedings of the IJCAI, F, 2018.
[7]	HOU M, LIU S, ZHOU J, et al. Extreme low-resolution activity recognition using a super-resolution-oriented generative adversarial network[J]. Micromachines, 2021, 12(6): 670.
[8]	JOLICOEUR-MARTINEAU A. The relativistic discriminator: a key element missing from standard GAN[J]. arXiv preprint arXiv:180700734, 2018.
[9]	KUEHNE H, JHUANG H, GARROTE E, et al. HMDB: A Large Video Database for Human Motion Recognition[J]. IEEE, 2011: 2556-2563.
[10]	SOOMRO K, ZAMIR A R, SHAH M. UCF101: A dataset of 101 human actions classes from videos in the wild[C]// CRCV-TR-12-01, University of Central Florida, 2012.
[11]	KARPATHY A, TODERICI G, SHETTY S, et al. Large-scale video classification with convolutional neural networks[C]// Proceedings of the Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, F, 2014.
[12]	SIMONYAN K, ZISSERMAN A. Two-stream convolutional networks for action recognition in videos[J]. Advances in neural information processing systems, 2014: 568-576.
[13]	WANG L, XIONG Y, WANG Z, et al. Temporal segment networks for action recognition in videos[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41(11): 2740-55.
[14]	TRAN D, BOURDEV L, FERGUS R, et al. Learning spatiotemporal features with 3d convolutional networks[C]// Proceedings of the Proceedings of the IEEE international conference on computer vision, F, 2015.
[15]	SUN L, JIA K, YEUNG D-Y, et al. Human action recognition using factorized spatio-temporal convolutional networks[C]// Proceedings of the Proceedings of the IEEE international conference on computer vision, F, 2015.
[16]	DIBA A, FAYYAZ M, SHARMA V, et al. Temporal 3d convnets: New architecture and transfer learning for video classification[J]. arXiv preprint arXiv:171108200, 2017.
[17]	RYOO M, KIM K, YANG H. Extreme low resolution activity recognition with multi-siamese embedding learning[C]// Proceedings of the Proceedings of the AAAI conference on artificial intelligence, F, 2018.
[18]	CHEN J, WU J, KONRAD J, et al. Semi-coupled two-stream fusion convnets for action recognition at extremely low resolutions[C]// Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), F, 2017. IEEE.
[19]	XU M, SHARGHI A, CHEN X, et al. Fully-coupled two-stream spatiotemporal networks for extremely low resolution action recognition[C]// Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), F, 2018.
[20]	RYOO M, ROTHROCK B, FLEMING C, et al. Privacy-preserving human activity recognition from extreme low resolution[C]// Proceedings of the Proceedings of the AAAI conference on artificial intelligence, F, 2017.
[21]	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11): 139-44.
[22]	TRAN D, WANG H, TORRESANI L, et al. A closer look at spatiotemporal convolutions for action recognition[C]// Proceedings of the Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, F, 2018.
[23]	ZHOU Y, SUN X, LUO C, et al. Spatiotemporal fusion in 3D CNNs: A probabilistic view[C]// Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, F, 2020.
[24]	CHEN Y, GE H, LIU Y, et al. Agpn: Action granularity pyramid network for video action recognition[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(8): 3912-3923.
[25]	HEDEGAARD L, IOSIFIDIS A. Continual 3D convolutional neural networks for real-time processing of videos[C]// Proceedings of the European Conference on Computer Vision, F, 2022.
[26]	KAHATAPITIYA K, RYOO M S. Coarse-fine networks for temporal activity detection in videos[C]// Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, F, 2021.
[27]	LI S, WANG Z, LIU Y, et al. FSformer: Fast-Slow Transformer for video action recognition[J]. Image and Vision Computing, 2023, 137.
[28]	HUANG Y, LU Z, SHAO Z, et al. Simultaneous denoising and super-resolution of optical coherence tomography images based on generative adversarial network[J]. Optics express, 2019, 27(9): 12289-307.
[29]	STROUD J, ROSS D, SUN C, et al. D3d: Distilled 3d networks for video action recognition[C]// Proceedings of the Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, F, 2020.
[30]	RAGHAV G, SAMIRA E K, VINCENT M, et al. The ”something something” video database for learning and evaluating visual common sense[J]. In ICCV, 2017, 1(5): 5843-5851.
[31]	KINGMA D P, BA J L. Adam: A method for stochastic optimization[J]. arXiv, 2014 https://doi.org/10.48550/arXiv.1412.6980.
[32]	XIE S, SUN C, HUANG J, et al. Rethinking spatiotemporal feature learning for video understanding[J]. arXiv preprint arXiv:171204851, 2017.
[33]	LIN J, GAN C, HAN S. Tsm: Temporal shift module for efficient video understanding[C]// Proceedings of the Proceedings of the IEEE/CVF international conference on computer vision, F, 2019.
[34]	JIANG B, YAN J, WANG M, et al. STM: SpatioTemporal and Motion Encoding for Action Recognition[Z]. 2019
[35]	LI Y, JI B, SHI X, et al. Tea: Temporal excitation and aggregation for action recognition[C]// Proceedings of the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, F, 2020.
[36]	WANG L, TONG Z, JI B, et al. Tdn: Temporal difference networks for efficient action recognition[C]// Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, F, 2021.
[37]	LIU Y, YUAN J, TU Z. Motion-driven visual tempo learning for video-based action recognition[J]. IEEE Transactions on Image Processing, 2022, 31: 4104-16.
[38]	LI K. Uniformer: Unified transformer for efficient spatial-temporal representation learning[C]// in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit, 2022: 2380-2390.
[39]	CARREIRA J, ZISSERMAN A. Quo vadis, action recognition? a new model and the kinetics dataset[C]// Proceedings of the proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, F, 2017.

相对平均的区分器模块	注意力机制的超级分辨率模块	精度
相对平均的区分器模块	注意力机制的超级分辨率模块	HMDB51	UCF101
×	×	78.7%	97%
√	×	79.8%	97.9%
×	√	79.1%	97.3%
√	√	80.1%	98.2%

方法	骨架网络	预训练	Top-1
S3D^[32]	Inception V2	Kinetics	75.9%
R(2+1)D^[22]	ResNet18	Kinetics	74.5%
TSM^[33]	Inception V2	Kinetics	73.2%
STM^[34] TEA^[35] TDN^[36] TCM^[37] D3D^[29] UniFormer-B^[38] ours	ResNet34 ResNet50 ResNet50 ResNet50 ResNet50 ResNet50 ResNet50	Kinetics Kinetics Kinetics Kinetics Kinetics Kinetics Kinetics	72.2% 73.3% 76.3% 77.5% 78.7% 79.6% 80.1%

方法	骨架网络	预训练	Top-1
S3D^[32]	Inception V2	Kinetics	96.8%
R(2+1)D^[22]	ResNet18	Kinetics	96.8%
TSM^[33]	Inception V2	Kinetics	96.0%
STM^[34] TEA^[35] TDN^[36] TCM^[37] D3D^[29] UniFormer-B^[38] Two-Stream I3D^[39] Ours	ResNet34 ResNet50 ResNet50 ResNet50 ResNet50 ResNet50 ResNet50 ResNet50	Kinetics Kinetics Kinetics Kinetics Kinetics Kinetics Kinetics Kinetics	96.2% 96.9% 97.4% 97.1% 97.0% 97.1% 97.8% 98.2%

方法	骨架网络	Param.	GFLOPs	Top1(Sth-Sth V1)
S3D^[32]	Inception V2	20.5M	78G	47.3%
TSM^[33]	Inception V2	24.3M	98G	49.7%
STM^[34] TEA^[35] TDN^[36] TCM^[37] Ours	ResNet34 ResNet50 ResNet50 ResNet50 ResNet50	24.0M 43.1M 35.4M 49.0M 33.0M	67×30G 65G 108G 105G 48G	50.7% 52.6% 55.1% 57.2% 60.5%
方法	骨架网络	Param.	GFLOPs	Top1(Sth-Sth V2)
TSM^[33]	Inception V2	35.3M	65×6G	66.6%
STM^[34] TEA^[35] TDN^[36] TCM^[37] UniFormer-B^[38] Ours	ResNet34 ResNet50 ResNet101 ResNet50 ResNet50 ResNet50	24.0M 24.5M 35.4M 49.0M 50.0M 33.0M	67×30G 65G 198G 105G 777G 48G	67.2% 67.9% 68.2% 67.8% 71.2% 72.6%