基于注意力和相对平均区分器的视频行为识别模型

doi:10.11871/jfdc.issn.2096-742X.2025.02.011

数据与计算发展前沿 ›› 2025, Vol. 7 ›› Issue (2): 109-119.

CSTR: 32002.14.jfdc.CN10-1649/TP.2025.02.011

doi: 10.11871/jfdc.issn.2096-742X.2025.02.011

基于注意力和相对平均区分器的视频行为识别模型

王岐军¹(),刘廷龙^2,^*()

1.大连港通信工程有限公司，辽宁大连 116001
2.大连工业大学信息技术中心，辽宁大连 116034

收稿日期:2024-09-17 出版日期:2025-04-20 发布日期:2025-04-23
通讯作者: 刘廷龙
作者简介:王岐军，研究生，高级工程师，国际项目经理（IPMA），硕士生导师，大连港通信工程有限公司总经理。长期从事港口软件、安全软件产品研发建设，主要研究方向是智能信息系统，智能信息处理。
负责论文初稿撰写和实验开发。
WANG Qijun, Master’s degree, Senior Engineer, International Project Manager (IPMA), Master tutor, is the General Manager of Dalian Port Communication Engineering Co., LTD. He has long been engaged in the development and construction of port software and security software products. His main research interests include intelligent information system and intelligent information processing.
In this paper, he is responsible for the paper drafting and experiment development.
E-mail: qjwang@cmhk.com|刘廷龙，大连工业大学信息技术中心，实验教师，长期从事人工智能，信息系统开发建设，主要研究方向是计算机视觉，智能信息系统。
负责制定论文框架和理论分析验证。
LIU Tinglong, is an experimental teacher at the Center for Information Technology of Dalian Polytechnic University,. He has long been engaged in artificial intelligence, information system development and construction. His main research interests include intelligent information system and intelligent information processing.
In this paper, he is responsible for formulating paper framework and theoretical analysis and verification.
E-mail: liutl@dlpu.edu.cn
基金资助:
辽宁省科技计划项目(2020JH2/10100032)

Video Action Recognition Model Based on Attention and Relative Average Discriminator

WANG Qijun¹(),LIU Tinglong^2,^*()

1. Dalian Port Communication Engineering Co., LTD., Dalian, Liaoning 116001, China
2. Center for Information Technology, Dalian Polytechnic University, Dalian, Liaoning 116034, China

Received:2024-09-17 Online:2025-04-20 Published:2025-04-23
Contact: LIU Tinglong

摘要/Abstract

摘要：

【目的】在视频行为识别任务中，如何获取更多的行为特征是研究的重点内容。视频行为特征包括时序特征和空间特征。如果模型无法抽取足够的时空特征，将严重影响视频行为识别结果。【方法】超分辨率技术主要作用是将低分辨率图片转换为高分辨率图片。已经有研究者将超级分辨率技术应用到了低分辨率视频行为识别领域，但方法主要从视频的整体出发进行视频帧图片分辨率的提升，对行为本身的特征关注度不足。为解决这个问题，本文提出了一种基于注意力机制和相对平均区分器的超级分辨率（ARADSP）网络模型方法。【结果】通过注意力机制关注视频中具有更多时空信息的行为特征，通过相对平均的区分器来提升数据质量和稳定性。最后在数据集HMDB51、UCF101和Something-Something V1&V2上进行实验，大量的实验结果表明本文的方法在视频行为识别中的有效性。

关键词: 注意力机制, 超级分辨率, 对抗生成网络, 相对平均区分器, 视频行为识别

Abstract:

[Objective] In the task of video action recognition, how to obtain more action features is a key research issue. Video action features include temporal and spatial features. If the model cannot extract enough spatiotemporal features, it will seriously affect the results of video action recognition. [Methods] Super-resolution technology mainly serves to convert low-resolution images into high-resolution images; some researchers have applied super-resolution technology to the field of low-resolution video action recognition. However, these methods mainly focus on enhancing the resolution of video frame images from the overall perspective of the video, paying insufficient attention to the features of the action itself. To address this issue, this paper proposes a super-resolution network model based on attention mechanisms and relative average discriminators (ARADSP). [Results] The proposed method focuses on action features in videos with more spatiotemporal information through attention mechanisms and improves data quality and stability through relative average discriminators. Finally, experiments are conducted on HMDB51, UCF101, and Something-Something V1&V2 datasets. Extensive experimental results demonstrate the effectiveness of the proposed method in video action recognition.

Key words: attention, supper-resolution, generative adversarial network, realistic average discriminator, low-resolution action recognition

王岐军,刘廷龙. 基于注意力和相对平均区分器的视频行为识别模型[J]. 数据与计算发展前沿, 2025, 7(2): 109-119.

WANG Qijun,LIU Tinglong. Video Action Recognition Model Based on Attention and Relative Average Discriminator[J]. Frontiers of Data and Computing, 2025, 7(2): 109-119, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2025.02.011.

图/表 9

图1

图2

表1

图3

图4

图5

表2

表3

表4

参考文献 39

[1]	吴建超, 王利民, 武港山. 视频群体行为识别综述[J]. 软件学报, 2023, 34(2): 964-984.
[2]	丁静, 舒祥波, 黄捧, 等. 基于多模态多粒度图卷积网络的老年人日常行为识别[J]. 软件学报, 2023, 34(5): 2350-64.
[3]	唐超, 王文剑, 李伟, 等. 基于多学习器协同训练模型的人体行为识别方法[J]. 软件学报, 2015, 26(11): 2939-50.
[4]	ATAER-CANSIZOGLU E, JONES M, ZHANG Z, et al. Verification of very low-resolution faces using an identity-preserving deep face super-resolution network[J]. arXiv preprint arXiv:190310974, 2019.
[5]	BAI Y, ZHANG Y, DING M, et al. Sod-mtgan: Small object detection via multi-task generative adversarial network[C]// Proceedings of the Proceedings of the European Conference on Computer Vision (ECCV), F, 2018.
[6]	WANG Z, YE M, YANG F, et al. Cascaded SR-GAN for scale-adaptive low resolution person re-identification[C]// Proceedings of the IJCAI, F, 2018.
[7]	HOU M, LIU S, ZHOU J, et al. Extreme low-resolution activity recognition using a super-resolution-oriented generative adversarial network[J]. Micromachines, 2021, 12(6): 670.
[8]	JOLICOEUR-MARTINEAU A. The relativistic discriminator: a key element missing from standard GAN[J]. arXiv preprint arXiv:180700734, 2018.
[9]	KUEHNE H, JHUANG H, GARROTE E, et al. HMDB: A Large Video Database for Human Motion Recognition[J]. IEEE, 2011: 2556-2563.
[10]	SOOMRO K, ZAMIR A R, SHAH M. UCF101: A dataset of 101 human actions classes from videos in the wild[C]// CRCV-TR-12-01, University of Central Florida, 2012.
[11]	KARPATHY A, TODERICI G, SHETTY S, et al. Large-scale video classification with convolutional neural networks[C]// Proceedings of the Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, F, 2014.
[12]	SIMONYAN K, ZISSERMAN A. Two-stream convolutional networks for action recognition in videos[J]. Advances in neural information processing systems, 2014: 568-576.
[13]	WANG L, XIONG Y, WANG Z, et al. Temporal segment networks for action recognition in videos[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41(11): 2740-55.
[14]	TRAN D, BOURDEV L, FERGUS R, et al. Learning spatiotemporal features with 3d convolutional networks[C]// Proceedings of the Proceedings of the IEEE international conference on computer vision, F, 2015.
[15]	SUN L, JIA K, YEUNG D-Y, et al. Human action recognition using factorized spatio-temporal convolutional networks[C]// Proceedings of the Proceedings of the IEEE international conference on computer vision, F, 2015.
[16]	DIBA A, FAYYAZ M, SHARMA V, et al. Temporal 3d convnets: New architecture and transfer learning for video classification[J]. arXiv preprint arXiv:171108200, 2017.
[17]	RYOO M, KIM K, YANG H. Extreme low resolution activity recognition with multi-siamese embedding learning[C]// Proceedings of the Proceedings of the AAAI conference on artificial intelligence, F, 2018.
[18]	CHEN J, WU J, KONRAD J, et al. Semi-coupled two-stream fusion convnets for action recognition at extremely low resolutions[C]// Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), F, 2017. IEEE.
[19]	XU M, SHARGHI A, CHEN X, et al. Fully-coupled two-stream spatiotemporal networks for extremely low resolution action recognition[C]// Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), F, 2018.
[20]	RYOO M, ROTHROCK B, FLEMING C, et al. Privacy-preserving human activity recognition from extreme low resolution[C]// Proceedings of the Proceedings of the AAAI conference on artificial intelligence, F, 2017.
[21]	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11): 139-44.
[22]	TRAN D, WANG H, TORRESANI L, et al. A closer look at spatiotemporal convolutions for action recognition[C]// Proceedings of the Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, F, 2018.
[23]	ZHOU Y, SUN X, LUO C, et al. Spatiotemporal fusion in 3D CNNs: A probabilistic view[C]// Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, F, 2020.
[24]	CHEN Y, GE H, LIU Y, et al. Agpn: Action granularity pyramid network for video action recognition[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(8): 3912-3923.
[25]	HEDEGAARD L, IOSIFIDIS A. Continual 3D convolutional neural networks for real-time processing of videos[C]// Proceedings of the European Conference on Computer Vision, F, 2022.
[26]	KAHATAPITIYA K, RYOO M S. Coarse-fine networks for temporal activity detection in videos[C]// Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, F, 2021.
[27]	LI S, WANG Z, LIU Y, et al. FSformer: Fast-Slow Transformer for video action recognition[J]. Image and Vision Computing, 2023, 137.
[28]	HUANG Y, LU Z, SHAO Z, et al. Simultaneous denoising and super-resolution of optical coherence tomography images based on generative adversarial network[J]. Optics express, 2019, 27(9): 12289-307.
[29]	STROUD J, ROSS D, SUN C, et al. D3d: Distilled 3d networks for video action recognition[C]// Proceedings of the Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, F, 2020.
[30]	RAGHAV G, SAMIRA E K, VINCENT M, et al. The ”something something” video database for learning and evaluating visual common sense[J]. In ICCV, 2017, 1(5): 5843-5851.
[31]	KINGMA D P, BA J L. Adam: A method for stochastic optimization[J]. arXiv, 2014 https://doi.org/10.48550/arXiv.1412.6980.
[32]	XIE S, SUN C, HUANG J, et al. Rethinking spatiotemporal feature learning for video understanding[J]. arXiv preprint arXiv:171204851, 2017.
[33]	LIN J, GAN C, HAN S. Tsm: Temporal shift module for efficient video understanding[C]// Proceedings of the Proceedings of the IEEE/CVF international conference on computer vision, F, 2019.
[34]	JIANG B, YAN J, WANG M, et al. STM: SpatioTemporal and Motion Encoding for Action Recognition[Z]. 2019
[35]	LI Y, JI B, SHI X, et al. Tea: Temporal excitation and aggregation for action recognition[C]// Proceedings of the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, F, 2020.
[36]	WANG L, TONG Z, JI B, et al. Tdn: Temporal difference networks for efficient action recognition[C]// Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, F, 2021.
[37]	LIU Y, YUAN J, TU Z. Motion-driven visual tempo learning for video-based action recognition[J]. IEEE Transactions on Image Processing, 2022, 31: 4104-16.
[38]	LI K. Uniformer: Unified transformer for efficient spatial-temporal representation learning[C]// in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit, 2022: 2380-2390.
[39]	CARREIRA J, ZISSERMAN A. Quo vadis, action recognition? a new model and the kinetics dataset[C]// Proceedings of the proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, F, 2017.

相对平均的区分器模块	注意力机制的超级分辨率模块	精度
相对平均的区分器模块	注意力机制的超级分辨率模块	HMDB51	UCF101
×	×	78.7%	97%
√	×	79.8%	97.9%
×	√	79.1%	97.3%
√	√	80.1%	98.2%

方法	骨架网络	预训练	Top-1
S3D^[32]	Inception V2	Kinetics	75.9%
R(2+1)D^[22]	ResNet18	Kinetics	74.5%
TSM^[33]	Inception V2	Kinetics	73.2%
STM^[34] TEA^[35] TDN^[36] TCM^[37] D3D^[29] UniFormer-B^[38] ours	ResNet34 ResNet50 ResNet50 ResNet50 ResNet50 ResNet50 ResNet50	Kinetics Kinetics Kinetics Kinetics Kinetics Kinetics Kinetics	72.2% 73.3% 76.3% 77.5% 78.7% 79.6% 80.1%

方法	骨架网络	预训练	Top-1
S3D^[32]	Inception V2	Kinetics	96.8%
R(2+1)D^[22]	ResNet18	Kinetics	96.8%
TSM^[33]	Inception V2	Kinetics	96.0%
STM^[34] TEA^[35] TDN^[36] TCM^[37] D3D^[29] UniFormer-B^[38] Two-Stream I3D^[39] Ours	ResNet34 ResNet50 ResNet50 ResNet50 ResNet50 ResNet50 ResNet50 ResNet50	Kinetics Kinetics Kinetics Kinetics Kinetics Kinetics Kinetics Kinetics	96.2% 96.9% 97.4% 97.1% 97.0% 97.1% 97.8% 98.2%

方法	骨架网络	Param.	GFLOPs	Top1(Sth-Sth V1)
S3D^[32]	Inception V2	20.5M	78G	47.3%
TSM^[33]	Inception V2	24.3M	98G	49.7%
STM^[34] TEA^[35] TDN^[36] TCM^[37] Ours	ResNet34 ResNet50 ResNet50 ResNet50 ResNet50	24.0M 43.1M 35.4M 49.0M 33.0M	67×30G 65G 108G 105G 48G	50.7% 52.6% 55.1% 57.2% 60.5%
方法	骨架网络	Param.	GFLOPs	Top1(Sth-Sth V2)
TSM^[33]	Inception V2	35.3M	65×6G	66.6%
STM^[34] TEA^[35] TDN^[36] TCM^[37] UniFormer-B^[38] Ours	ResNet34 ResNet50 ResNet101 ResNet50 ResNet50 ResNet50	24.0M 24.5M 35.4M 49.0M 50.0M 33.0M	67×30G 65G 198G 105G 777G 48G	67.2% 67.9% 68.2% 67.8% 71.2% 72.6%

基于注意力和相对平均区分器的视频行为识别模型

Video Action Recognition Model Based on Attention and Relative Average Discriminator

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 39

相关文章 10

编辑推荐

Metrics

本文评价

[1]	李勇,任勇毛,殷卓然,周旭. 一种基于深度学习的轻量化流量识别模型[J]. 数据与计算发展前沿, 2025, 7(2): 3-11.
[2]	晏直誉, 茹一伟, 孙福鹏, 孙哲南. 基于主动感知机制的视频行为识别方法研究[J]. 数据与计算发展前沿, 2024, 6(5): 66-79.
[3]	李浩鹏, 周琬婷, 陈玉, 张曼. 基于域无关循环生成对抗网络的跨模态医学影像生成[J]. 数据与计算发展前沿, 2024, 6(2): 80-88.
[4]	申志豪, 李娜, 尹世豪, 杜一, 胡良霖. 基于TPA-Transformer的机票价格预测[J]. 数据与计算发展前沿, 2023, 5(6): 115-125.
[5]	王子元, 王国中. 改进的轻量级YOLOv5算法在行人检测的应用[J]. 数据与计算发展前沿, 2023, 5(6): 161-172.
[6]	张蓉, 刘渊. 适用于方面级情感分析的多级数据增强方法[J]. 数据与计算发展前沿, 2023, 5(5): 140-153.
[7]	张晓帆, 孙海春, 李欣. 融合多层注意力机制与BiLSTM的知识图谱补全算法研究[J]. 数据与计算发展前沿, 2023, 5(3): 123-137.
[8]	刘琦玮,李俊,顾蓓蓓,赵泽方. TSAIE：图像增强文本的多模态情感分析模型[J]. 数据与计算发展前沿, 2022, 4(3): 131-140.
[9]	肖楠,周明珠,邢军,罗泽,李晓辉. 基于高分辨率网络和注意力机制的真伪卷烟包装鉴别[J]. 数据与计算发展前沿, 2021, 3(5): 118-129.
[10]	冷佳旭,刘莹. 基于深度学习的小目标检测与识别[J]. 数据与计算发展前沿, 2020, 2(2): 120-135.