一种用于面部视频抑郁水平评估的多尺度时间差分网络

doi:10.11871/jfdc.issn.2096-742X.2026.04.011

数据与计算发展前沿 ›› 2026, Vol. 8 ›› Issue (4): 149-161.

CSTR: 32002.14.jfdc.CN10-1649/TP.2026.04.011

doi: 10.11871/jfdc.issn.2096-742X.2026.04.011

一种用于面部视频抑郁水平评估的多尺度时间差分网络

张梓豪¹(),张小瑞^2,^*(),孙伟³,周石煜¹

¹ 南京信息工程大学，软件学院，江苏南京 210044
² 南京工业大学，计算机与信息工程学院，江苏南京 211816
³ 南京信息工程大学，自动化学院，江苏南京 210044

收稿日期:2026-02-24 出版日期:2026-08-20 发布日期:2026-08-21
通讯作者: *张小瑞（E-mail: zxr365@126.com）
作者简介:张梓豪，南京信息工程大学，硕士研究生，主要研究方向为深度学习与视频识别。
本文中负责实验设计及实施、论文撰写。
ZHANG Zihao is a master’s student at Nanjing University of Information Science and Technology. His research interests include deep learning and video recognition.
In this paper, he is responsible for the experimental design and implementation, as well as the manuscript writing.
E-mail: zzh920828559@163.com|张小瑞，南京工业大学计算机与信息工程学院，教授，主要研究方向包括数字取证、图像处理、计算机视觉、虚拟现实及人机交互等。
本文中负责研究指导、论文修订与校对。
ZHANG Xiaorui is a professor at the School of Computer Science and Information Engineering, Nanjing Tech University. Her research interests include digital forensics, image processing, computer vision, virtual reality, and human-computer interaction.
In this paper, she is mainly responsible for research supervision, manuscript revision, and proofreading.
E-mail: zxr365@126.com
基金资助:
国家自然科学基金(62272236);江苏省自然科学基金(BK20201136);江苏省自然科学基金(BK20191401)

A Multi-Scale Temporal Difference Network for Depression Level Assessment from Facial Video

ZHANG Zihao¹(),ZHANG Xiaorui^2,^*(),SUN Wei³,ZHOU Shiyu¹

¹ School of Software, Nanjing University of Information Science and Technology, Nanjing, Jiangsu 210044, China
² College of Computer and information Engineering, Nanjing Tech University, Nanjing, Jiangsu 211816, China
³ School of Automation, Nanjing University of Information Science and Technology, Nanjing, Jiangsu 210044, China

Received:2026-02-24 Online:2026-08-20 Published:2026-08-21

摘要/Abstract

摘要：

【目的】 针对现有基于视频的抑郁检测方法计算开销过高的问题，本文提出了一种以兼顾预测精度与计算开销为目的的多尺度时间差分网络。【方法】 本文通过关键帧采样块定位表情突变区间并选取关键帧，在减少数据冗余的同时保留更多可用特征，通过时间差分块实现从短期到长期的层次化动态建模，通过时空融合块构建混合序列以对齐时空特征，并结合伪3D卷积实现时空信息的高效融合，为网络提供更全面的抑郁线索。【结果】 在两个公共数据集上进行相关实验，所设计的多尺度时间差分网络参数量仅为1.03M，浮点运算次数低至1.97 GFLOPs，在MAE和RMSE两个指标上也与其它先进抑郁水平评估方法保持极具竞争力的性能。【结论】 实验表明，本文提出的方法在极大地降低计算复杂度的同时很好地维持了精度，为抑郁水平任务提供了一种新思路。

关键词: 深度学习, 抑郁检测, 时间差分, 多尺度, 面部特征

Abstract:

[Objective] To address the high computational overhead in existing video-based depression assessment methods, this study proposes a Multi-scale Temporal Difference Network (MTDN) designed to balance prediction accuracy and computational efficiency. [Methods] The framework integrates three novel components: The Key Frame Sampling Block computes temporal differences between adjacent frames to quantify inter-frame variations, while a threshold-based binarization strategy identifies expression-change intervals and selects salient frames to reduce redundancy. The Temporal Difference Block, composed of long-term and short-term Dynamic Modeling Submodules, employs bidirectional differencing combined with multi-scale feature alignment to capture dynamics from micro-expressions to sustained emotional shifts. Additionally, the Spatial-Temporal Fusion Block constructs mixed sequences and applies Pseudo-3D convolution to align spatial and temporal feature maps, integrating spatiotemporal information critical for detecting depression cues. [Results] Experiments on two datasets demonstrate that MTDN achieves state-of-the-art efficiency with only 1.03M parameters and 1.97 GFLOPs. It also achieves highly competitive performance compared with advanced depression assessment methods in terms of MAE and RMSE. [Conclusions] This study establishes a new trade-off frontier for depression assessment, enabling deployment on resource-constrained devices without sacrificing diagnostic accuracy. This approach provides actionable insights for developing lightweight clinical decision support systems.

Key words: deep learning, depression detection, temporal difference, multi-scale, facial features

张梓豪, 张小瑞, 孙伟, 周石煜. 一种用于面部视频抑郁水平评估的多尺度时间差分网络[J]. 数据与计算发展前沿, 2026, 8(4): 149-161.

ZHANG Zihao, ZHANG Xiaorui, SUN Wei, ZHOU Shiyu. A Multi-Scale Temporal Difference Network for Depression Level Assessment from Facial Video[J]. Frontiers of Data and Computing, 2026, 8(4): 149-161, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2026.04.011.

图/表 12

图1

图2

图3

图4

表1

表2

图5

图6

表3

表4

表5

图7

参考文献 39

[1]	DEPRESSION W H O. Other common mental disorders: global health estimates[J]. Geneva: World Health Organization, 2017, 24(1): 1-24.
[2]	JAMES S L, ABATE D, ABATE K H, et al. Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990-2017: a systematic analysis for the Global Burden of Disease Study 2017[J]. The lancet, 2018, 392(10159): 1789-1858. doi: 10.1016/S0140-6736(18)32279-7
[3]	VAHIA V N. Diagnostic and statistical manual of mental disorders 5: A quick glance[J]. Indian journal of psychiatry, 2013, 55(3): 220-223. doi: 10.4103/0019-5545.117131 pmid: 24082241
[4]	NODA T, YOSHIDA S, MATSUDA T, et al. Frontal and right temporal activations correlate negatively with depression severity during verbal fluency task: a multi-channel near-infrared spectroscopy study[J]. Journal of psychiatric research, 2012, 46(7): 905-912. doi: 10.1016/j.jpsychires.2012.04.001 pmid: 22572569
[5]	HAWTON K, I COMABELLA C C, HAW C, et al. Risk factors for suicide in individuals with depression: a systematic review[J]. Journal of affective disorders, 2013, 147(1-3): 17-28. doi: 10.1016/j.jad.2013.01.004 pmid: 23411024
[6]	PAMPOUCHIDOU A, SIMOS P G, MARIAS K, et al. Automatic assessment of depression based on visual cues: A systematic review[J]. IEEE Transactions on Affective Computing, 2017, 10(4): 445-470. doi: 10.1109/T-AFFC.5165369
[7]	Zhang Z, Meng Q, Jin L C, et al. A novel EEG-based graph convolution network for depression detection: incorporating secondary subject partitioning and attention mechanism[J]. Expert Systems with Applications, 2024, 239(1): 122356. doi: 10.1016/j.eswa.2023.122356
[8]	ZHONG J, WU Y, LIU H, et al. Soft fusion of channel information in depression detection using functional near-infrared spectroscopy[J]. Information Processing & Management, 2025, 62(3): 104003. doi: 10.1016/j.ipm.2024.104003
[9]	Kotoula V, Evans J W, Punturieri C E, et al. The use of functional magnetic resonance imaging (fMRI) in clinical trials and experimental research studies for depression[J]. Frontiers in Neuroimaging, 2023, 2(1): 1110258. doi: 10.3389/fnimg.2023.1110258
[10]	SONG S, LUO Y, TUMER T, et al. Loss relaxation strategy for noisy facial video-based automatic depression recognition[J]. ACM Transactions on Computing for Healthcare, 2024, 5(2): 1-24.
[11]	HE L, JIANG D, SAHLI H. Multimodal depression recognition with dynamic visual and audio cues[C]// 2015 International conference on affective computing and intelligent interaction (ACII), IEEE, 2015: 260-266.
[12]	CUMMINS N, SETHU V, JOSHI J, et al. Diagnosis of Depression by Behavioural Signals: A Multimodal Approach[C]// ACM International Workshop on Audio/Visual Emotion Challenge, Association for Computing Machinery (ACM), 2013: 11-20.
[13]	DHALL A, GOECKE R. A temporally piece-wise fisher vector approach for depression analysis[C]// 2015 International conference on affective computing and intelligent interaction (ACII), IEEE, 2015: 255-259.
[14]	JAN A, MENG H, GAUS Y F A, et al. Automatic depression scale prediction using facial expression dynamics and regression[C]// Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, 2014: 73-80.
[15]	OTTE C, GOLD S M, PENNINX B W, et al. Major depressive disorder[J]. Nature reviews Disease primers, 2016, 2(1): 1-20.
[16]	SHANG Y, PAN Y, JIANG X, et al. LQGDNet: A local quaternion and global deep network for facial depression recognition[J]. IEEE transactions on affective computing, 2021, 14(3): 2557-2563. doi: 10.1109/TAFFC.2021.3139651
[17]	DE MELO W C, GRANGER E, HADID A. A deep multiscale spatiotemporal network for assessing depression from facial dynamics[J]. IEEE transactions on affective computing, 2020, 13(3): 1581-1592. doi: 10.1109/TAFFC.2020.3021755
[18]	HADjI I, WILDES R P. A new large scale dynamic texture dataset with application to convnet understanding[C]// Proceedings of the European Conference on Computer Vision (ECCV), 2018: 320-335.
[19]	DE MELO W C, GRANGER E, LOPEZ M B. MDN: A deep maximization-differentiation network for spatio-temporal depression detection[J]. IEEE transactions on affective computing, 2021, 14(1): 578-590. doi: 10.1109/TAFFC.2021.3072579
[20]	MENG H, HUANG D, WANG H, et al. Depression recognition based on dynamic facial and vocal expression features using partial least square regression[C]// Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge, 2013: 21-30.
[21]	WANG Y, MA J, HAO B, et al. Automatic depression detection via facial expressions using multiple instance learning[C]// 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), IEEE, 2020: 1933-1936.
[22]	TRAN D, WANG H, TORRESANI L, et al. A closer look at spatiotemporal convolutions for action recognition[C]// Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2018: 6450-6459.
[23]	NIU M, ZHAO Z, TAO J, et al. Dual attention and element recalibration networks for automatic depression level prediction[J]. IEEE Transactions on Affective Computing, 2022, 14(3): 1954-1965. doi: 10.1109/TAFFC.2022.3177737
[24]	VALSTAR M, SCHULLER B, SMITH K, et al. Avec 2013: the continuous audio/visual emotion and depression recognition challenge[C]// Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge, 2013: 3-10.
[25]	VALSTAR M, SCHULLER B, SMITH K, et al. Avec 2014: 3d dimensional affect and depression recognition challenge[C]// Proceedings of the 4th international workshop on audio/visual emotion challenge, 2014: 3-10.
[26]	AL JAZAERY M, GUO G. Video-based depression level analysis by encoding deep spatiotemporal features[J]. IEEE Transactions on Affective Computing, 2018, 12(1): 262-268. doi: 10.1109/T-AFFC.5165369
[27]	NIU M, TAO J, LIU B, et al. Multimodal spatiotemporal representation for automatic depression level detection[J]. IEEE transactions on affective computing, 2020, 14(1): 294-307. doi: 10.1109/TAFFC.2020.3031345
[28]	WANG L, XIONG Y, WANG Z, et al. Temporal segment networks: Towards good practices for deep action recognition[C]// European conference on computer vision, Cham: Springer International Publishing, 2016: 20-36.
[29]	UDDIN M A, JOOLEE J B, LEE Y K. Depression level prediction using deep spatiotemporal features and multilayer bi-ltsm[J]. IEEE Transactions on Affective Computing, 2020, 13(2): 864-870. doi: 10.1109/TAFFC.2020.2970418
[30]	CARREIRA J, ZISSERMAN A. Quo vadis, action recognition? a new model and the kinetics dataset[C]// proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 6299-6308.
[31]	DE MELO W C, GRANGER E, LOPEZ M B. Facial expression analysis using decomposed multiscale spatiotemporal networks[J]. Expert Systems with Applications, 2024, 236(1): 121276. doi: 10.1016/j.eswa.2023.121276
[32]	HE L, LI Z, TIWARI P, et al. LSCAformer: Long and short-term cross-attention-aware transformer for depression recognition from video sequences[J]. Biomedical Signal Processing and Control, 2024, 98(1): 106767. doi: 10.1016/j.bspc.2024.106767
[33]	ZHANG S, ZHANG X, ZHAO X, et al. MTDAN: A lightweight multi-scale temporal difference attention networks for automated video depression detection[J]. IEEE transactions on affective computing, 2023, 15(3): 1078-1089. doi: 10.1109/TAFFC.2023.3312263
[34]	ZHU Y, SHANG Y, SHAO Z, et al. Automated depression diagnosis based on deep networks to encode facial appearance and dynamics[J]. IEEE Transactions on Affective Computing, 2017, 9(4): 578-584. doi: 10.1109/T-AFFC.5165369
[35]	DE MELO W C, GRANGER E, HADID A. Depression detection based on deep distribution learning[C]// 2019 IEEE international conference on image processing (ICIP), IEEE, 2019: 4544-4548.
[36]	CHEN Q, CHATURVEDI I, JI S, et al. Sequential fusion of facial appearance and dynamics for depression recognition[J]. Pattern Recognition Letters, 2021, 150(1): 115-121. doi: 10.1016/j.patrec.2021.07.005
[37]	DE MELO W C, GRANGER E, HADID A. Combining global and local convolutional 3d networks for detecting depression from facial expressions[C]// 2019 14th ieee international conference on automatic face & gesture recognition (fg 2019), IEEE, 2019: 1-8.
[38]	PAN Y, SHANG Y, LIU T, et al. Spatial-temporal attention network for depression recognition from facial videos[J]. Expert systems with applications, 2024, 237(1): 121410. doi: 10.1016/j.eswa.2023.121410
[39]	HE L, ZHAO J, ZHANG J, et al. LMTformer: facial depression recognition with lightweight multi-scale transformer from videos.[J]. Applied Intelligence, 2025, 55(2): 195. doi: 10.1007/s10489-024-05908-x

方法	AVEC2013		AVEC2014
方法	RMSE	MAE	RMSE	MAE
TSN	8.37	6.61	8.46	6.77
Spatial	8.89	7.02	9.31	7.47
Temporal	8.93	7.08	9.17	7.25
Sbranch	8.95	6.77	9.05	6.98
Lbranch	9.08	6.74	9.26	7.11
Features-level Fusion	8.54	6.86	8.84	6.86
MTDN	8.04	6.32	8.21	6.57

采样数量	AVEC2013		AVEC2014
采样数量	RMSE	MAE	RMSE	MAE
8	8.44	6.57	8.93	7.07
16	8.04	6.32	8.21	6.57
24	8.61	6.70	8.57	6.84

方法	RMSE	MAE
LPQ+SVR^[24]	13.61	10.88
MHH+PLS^[20]	11.19	9.14
LBP-TOP+SVR^[13]	8.91	7.08
DCNNs^[34]	9.82	7.58
RNN-C3D^[26]	9.28	7.37
Global-Local C3D^[37]	8.26	6.40
ResNet50^[35]	8.25	6.30
MSN^[17]	7.90	5.98
Bi-LSTM^[29]	8.93	7.04
LQGDNet^[16]	8.20	6.38
MDN-152^[19]	7.55	6.24
DAER^[23]	8.13	6.28
DMSN^[31]	7.66	6.14
MTDAN^[33]	8.08	6.14
STA-DRN^[38]	7.98	6.15
LSCAformer^[32]	7.69	5.89
LMTformer^[39]	7.75	6.12
Ours	8.04	6.32

方法	RMSE	MAE
LGBP-TOP+SVR^[25]	10.86	8.86
MHH+PLS^[20]	10.50	8.44
LBP-TOP+SVR^[13]	8.91	7.08
DCNNs^[34]	9.55	7.47
RNN-C3D^[26]	9.20	7.22
Global-Local C3D^[37]	8.31	6.59
ResNet50^[35]	8.23	6.15
MSN^[17]	7.61	5.82
Bi-LSTM^[29]	8.78	6.86
LQGDNet^[16]	7.84	6.08
MDN-152^[19]	7.65	6.06
DAER^[23]	8.07	6.14
DMSN^[31]	7.50	5.69
MTDAN^[33]	7.93	6.35
STA-DRN^[38]	7.75	6.00
LSCAformer^[32]	7.55	5.91
LMTformer^[39]	7.97	6.05
Ours	8.21	6.57

方法	AVEC2013		AVEC2014		Param/M	FLOPs/G
方法	RMSE	MAE	RMSE	MAE	Param/M	FLOPs/G
I3D	8.66	6.64	8.55	6.36	≈13	6.99
T3D	8.75	6.76	8.55	6.54	≈68	51.64
ResNet-50	8.25	6.30	8.23	6.15	≈25	3.80
MDN-50	8.13	6.39	8.16	6.45	≈21	7.40
MDN-152	7.55	6.24	7.65	6.06	≈52	13.36
Ours	8.04	6.32	8.21	6.57	1.03	1.97

一种用于面部视频抑郁水平评估的多尺度时间差分网络

A Multi-Scale Temporal Difference Network for Depression Level Assessment from Facial Video

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 12

参考文献 39

相关文章 15

编辑推荐

Metrics

本文评价

[1]	林予松, 梁振羽, 赵哲. PMDeUNet：一个用于高效分割腹部医学图像的轻量级网络模型[J]. 数据与计算发展前沿, 2026, 8(4): 162-178.
[2]	张思扬, 胡林. 融合标签语义的农业科学数据层次多标签分类方法研究[J]. 数据与计算发展前沿, 2026, 8(4): 246-256.
[3]	琚子政, 陈鹏, 隋晋光, 朱隆昇. 基于多尺度时空图融合的警情预测模型[J]. 数据与计算发展前沿, 2026, 8(2): 154-170.
[4]	周玉明, 张一鸣, 刘圆圆, 黄山. 遥感卫星图像船舶检测样本数据集研究综述[J]. 数据与计算发展前沿, 2026, 8(2): 215-226.
[5]	王兆滨, 王睿, 吕永科, 张耀南. 基于自适应语义连接和感知注意力的沙漠分割方法[J]. 数据与计算发展前沿, 2026, 8(2): 25-39.
[6]	韩立钦, 李龙园, 王钰康, 张耀南, 常盟盟, 潘清元. 基于频域引导特征的泥石流灾后遥感影像语义分割网络构建研究[J]. 数据与计算发展前沿, 2026, 8(2): 54-65.
[7]	潘语泉,袁得嵛,贾源,王安然. 基于融合特征的VGAT-VGAN跨社交网络身份关联算法[J]. 数据与计算发展前沿, 2026, 8(1): 103-118.
[8]	蔡毅,王晓宾,陈蕊丽,韩珣. 基于笔迹的书写者性别与年龄检测研究综述[J]. 数据与计算发展前沿, 2026, 8(1): 129-147.
[9]	万萌, 何洪林, 任小丽, 聂宁明, 曹荣强, 王宗国, 李凯, 王晓光, 王彦棡, 王珏, 高超. 基于工作流的陆地生态系统碳循环实时同化预测系统[J]. 数据与计算发展前沿, 2026, 8(1): 168-182.
[10]	邓绎如,何洪波,王英,王闰强. 一种基于情感分析的网络舆论倾向性检测方法[J]. 数据与计算发展前沿, 2026, 8(1): 91-102.
[11]	杨沁蒙, 聂宁明, 周纯葆, 王彦棡. 基于深度学习的Taylor杆碰撞数值模拟算法[J]. 数据与计算发展前沿, 2025, 7(6): 101-110.
[12]	周法国, 刘芳, 王彦棡, 王珏, 于淼, 李顺德, 周纯葆, 王婧, 杨沁蒙. 面向国产超算的深度学习框架算子的移植与适配[J]. 数据与计算发展前沿, 2025, 7(6): 136-148.
[13]	令狐荣微, 张瑜, 石元泉, 杨玉军. 基于多特征融合的PE恶意代码检测与分类研究[J]. 数据与计算发展前沿, 2025, 7(6): 77-91.
[14]	辛昱杭,王琦祎,孙婧,赵春燕,刘雨佳,梁雪,陈杰. 基于雷达回波外推的短临降水预报模型TrajCast在国产加速器上的实践[J]. 数据与计算发展前沿, 2025, 7(5): 113-122.
[15]	王鹏,杨霄凤,何忠臣,杜君. 基于浅层-深层卷积循环神经网络的多光谱遥感图像全色锐化方法[J]. 数据与计算发展前沿, 2025, 7(5): 138-152.