A Multi-Scale Temporal Difference Network for Depression Level Assessment from Facial Video

doi:10.11871/jfdc.issn.2096-742X.2026.04.011

Abstract

Abstract:

[Objective] To address the high computational overhead in existing video-based depression assessment methods, this study proposes a Multi-scale Temporal Difference Network (MTDN) designed to balance prediction accuracy and computational efficiency. [Methods] The framework integrates three novel components: The Key Frame Sampling Block computes temporal differences between adjacent frames to quantify inter-frame variations, while a threshold-based binarization strategy identifies expression-change intervals and selects salient frames to reduce redundancy. The Temporal Difference Block, composed of long-term and short-term Dynamic Modeling Submodules, employs bidirectional differencing combined with multi-scale feature alignment to capture dynamics from micro-expressions to sustained emotional shifts. Additionally, the Spatial-Temporal Fusion Block constructs mixed sequences and applies Pseudo-3D convolution to align spatial and temporal feature maps, integrating spatiotemporal information critical for detecting depression cues. [Results] Experiments on two datasets demonstrate that MTDN achieves state-of-the-art efficiency with only 1.03M parameters and 1.97 GFLOPs. It also achieves highly competitive performance compared with advanced depression assessment methods in terms of MAE and RMSE. [Conclusions] This study establishes a new trade-off frontier for depression assessment, enabling deployment on resource-constrained devices without sacrificing diagnostic accuracy. This approach provides actionable insights for developing lightweight clinical decision support systems.

Key words: deep learning, depression detection, temporal difference, multi-scale, facial features

ZHANG Zihao, ZHANG Xiaorui, SUN Wei, ZHOU Shiyu. A Multi-Scale Temporal Difference Network for Depression Level Assessment from Facial Video[J]. Frontiers of Data and Computing, 2026, 8(4): 149-161, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2026.04.011.

Figures/Tables 12

Fig.1

Fig.2

Fig.3

Fig.4

Table 1

Table 2

Fig.5

Fig.6

Table 3

Table 4

Table 5

Fig.7

References 39

[1]	DEPRESSION W H O. Other common mental disorders: global health estimates[J]. Geneva: World Health Organization, 2017, 24(1): 1-24.
[2]	JAMES S L, ABATE D, ABATE K H, et al. Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990-2017: a systematic analysis for the Global Burden of Disease Study 2017[J]. The lancet, 2018, 392(10159): 1789-1858. doi: 10.1016/S0140-6736(18)32279-7
[3]	VAHIA V N. Diagnostic and statistical manual of mental disorders 5: A quick glance[J]. Indian journal of psychiatry, 2013, 55(3): 220-223. doi: 10.4103/0019-5545.117131 pmid: 24082241
[4]	NODA T, YOSHIDA S, MATSUDA T, et al. Frontal and right temporal activations correlate negatively with depression severity during verbal fluency task: a multi-channel near-infrared spectroscopy study[J]. Journal of psychiatric research, 2012, 46(7): 905-912. doi: 10.1016/j.jpsychires.2012.04.001 pmid: 22572569
[5]	HAWTON K, I COMABELLA C C, HAW C, et al. Risk factors for suicide in individuals with depression: a systematic review[J]. Journal of affective disorders, 2013, 147(1-3): 17-28. doi: 10.1016/j.jad.2013.01.004 pmid: 23411024
[6]	PAMPOUCHIDOU A, SIMOS P G, MARIAS K, et al. Automatic assessment of depression based on visual cues: A systematic review[J]. IEEE Transactions on Affective Computing, 2017, 10(4): 445-470. doi: 10.1109/T-AFFC.5165369
[7]	Zhang Z, Meng Q, Jin L C, et al. A novel EEG-based graph convolution network for depression detection: incorporating secondary subject partitioning and attention mechanism[J]. Expert Systems with Applications, 2024, 239(1): 122356. doi: 10.1016/j.eswa.2023.122356
[8]	ZHONG J, WU Y, LIU H, et al. Soft fusion of channel information in depression detection using functional near-infrared spectroscopy[J]. Information Processing & Management, 2025, 62(3): 104003. doi: 10.1016/j.ipm.2024.104003
[9]	Kotoula V, Evans J W, Punturieri C E, et al. The use of functional magnetic resonance imaging (fMRI) in clinical trials and experimental research studies for depression[J]. Frontiers in Neuroimaging, 2023, 2(1): 1110258. doi: 10.3389/fnimg.2023.1110258
[10]	SONG S, LUO Y, TUMER T, et al. Loss relaxation strategy for noisy facial video-based automatic depression recognition[J]. ACM Transactions on Computing for Healthcare, 2024, 5(2): 1-24.
[11]	HE L, JIANG D, SAHLI H. Multimodal depression recognition with dynamic visual and audio cues[C]// 2015 International conference on affective computing and intelligent interaction (ACII), IEEE, 2015: 260-266.
[12]	CUMMINS N, SETHU V, JOSHI J, et al. Diagnosis of Depression by Behavioural Signals: A Multimodal Approach[C]// ACM International Workshop on Audio/Visual Emotion Challenge, Association for Computing Machinery (ACM), 2013: 11-20.
[13]	DHALL A, GOECKE R. A temporally piece-wise fisher vector approach for depression analysis[C]// 2015 International conference on affective computing and intelligent interaction (ACII), IEEE, 2015: 255-259.
[14]	JAN A, MENG H, GAUS Y F A, et al. Automatic depression scale prediction using facial expression dynamics and regression[C]// Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, 2014: 73-80.
[15]	OTTE C, GOLD S M, PENNINX B W, et al. Major depressive disorder[J]. Nature reviews Disease primers, 2016, 2(1): 1-20.
[16]	SHANG Y, PAN Y, JIANG X, et al. LQGDNet: A local quaternion and global deep network for facial depression recognition[J]. IEEE transactions on affective computing, 2021, 14(3): 2557-2563. doi: 10.1109/TAFFC.2021.3139651
[17]	DE MELO W C, GRANGER E, HADID A. A deep multiscale spatiotemporal network for assessing depression from facial dynamics[J]. IEEE transactions on affective computing, 2020, 13(3): 1581-1592. doi: 10.1109/TAFFC.2020.3021755
[18]	HADjI I, WILDES R P. A new large scale dynamic texture dataset with application to convnet understanding[C]// Proceedings of the European Conference on Computer Vision (ECCV), 2018: 320-335.
[19]	DE MELO W C, GRANGER E, LOPEZ M B. MDN: A deep maximization-differentiation network for spatio-temporal depression detection[J]. IEEE transactions on affective computing, 2021, 14(1): 578-590. doi: 10.1109/TAFFC.2021.3072579
[20]	MENG H, HUANG D, WANG H, et al. Depression recognition based on dynamic facial and vocal expression features using partial least square regression[C]// Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge, 2013: 21-30.
[21]	WANG Y, MA J, HAO B, et al. Automatic depression detection via facial expressions using multiple instance learning[C]// 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), IEEE, 2020: 1933-1936.
[22]	TRAN D, WANG H, TORRESANI L, et al. A closer look at spatiotemporal convolutions for action recognition[C]// Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2018: 6450-6459.
[23]	NIU M, ZHAO Z, TAO J, et al. Dual attention and element recalibration networks for automatic depression level prediction[J]. IEEE Transactions on Affective Computing, 2022, 14(3): 1954-1965. doi: 10.1109/TAFFC.2022.3177737
[24]	VALSTAR M, SCHULLER B, SMITH K, et al. Avec 2013: the continuous audio/visual emotion and depression recognition challenge[C]// Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge, 2013: 3-10.
[25]	VALSTAR M, SCHULLER B, SMITH K, et al. Avec 2014: 3d dimensional affect and depression recognition challenge[C]// Proceedings of the 4th international workshop on audio/visual emotion challenge, 2014: 3-10.
[26]	AL JAZAERY M, GUO G. Video-based depression level analysis by encoding deep spatiotemporal features[J]. IEEE Transactions on Affective Computing, 2018, 12(1): 262-268. doi: 10.1109/T-AFFC.5165369
[27]	NIU M, TAO J, LIU B, et al. Multimodal spatiotemporal representation for automatic depression level detection[J]. IEEE transactions on affective computing, 2020, 14(1): 294-307. doi: 10.1109/TAFFC.2020.3031345
[28]	WANG L, XIONG Y, WANG Z, et al. Temporal segment networks: Towards good practices for deep action recognition[C]// European conference on computer vision, Cham: Springer International Publishing, 2016: 20-36.
[29]	UDDIN M A, JOOLEE J B, LEE Y K. Depression level prediction using deep spatiotemporal features and multilayer bi-ltsm[J]. IEEE Transactions on Affective Computing, 2020, 13(2): 864-870. doi: 10.1109/TAFFC.2020.2970418
[30]	CARREIRA J, ZISSERMAN A. Quo vadis, action recognition? a new model and the kinetics dataset[C]// proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 6299-6308.
[31]	DE MELO W C, GRANGER E, LOPEZ M B. Facial expression analysis using decomposed multiscale spatiotemporal networks[J]. Expert Systems with Applications, 2024, 236(1): 121276. doi: 10.1016/j.eswa.2023.121276
[32]	HE L, LI Z, TIWARI P, et al. LSCAformer: Long and short-term cross-attention-aware transformer for depression recognition from video sequences[J]. Biomedical Signal Processing and Control, 2024, 98(1): 106767. doi: 10.1016/j.bspc.2024.106767
[33]	ZHANG S, ZHANG X, ZHAO X, et al. MTDAN: A lightweight multi-scale temporal difference attention networks for automated video depression detection[J]. IEEE transactions on affective computing, 2023, 15(3): 1078-1089. doi: 10.1109/TAFFC.2023.3312263
[34]	ZHU Y, SHANG Y, SHAO Z, et al. Automated depression diagnosis based on deep networks to encode facial appearance and dynamics[J]. IEEE Transactions on Affective Computing, 2017, 9(4): 578-584. doi: 10.1109/T-AFFC.5165369
[35]	DE MELO W C, GRANGER E, HADID A. Depression detection based on deep distribution learning[C]// 2019 IEEE international conference on image processing (ICIP), IEEE, 2019: 4544-4548.
[36]	CHEN Q, CHATURVEDI I, JI S, et al. Sequential fusion of facial appearance and dynamics for depression recognition[J]. Pattern Recognition Letters, 2021, 150(1): 115-121. doi: 10.1016/j.patrec.2021.07.005
[37]	DE MELO W C, GRANGER E, HADID A. Combining global and local convolutional 3d networks for detecting depression from facial expressions[C]// 2019 14th ieee international conference on automatic face & gesture recognition (fg 2019), IEEE, 2019: 1-8.
[38]	PAN Y, SHANG Y, LIU T, et al. Spatial-temporal attention network for depression recognition from facial videos[J]. Expert systems with applications, 2024, 237(1): 121410. doi: 10.1016/j.eswa.2023.121410
[39]	HE L, ZHAO J, ZHANG J, et al. LMTformer: facial depression recognition with lightweight multi-scale transformer from videos.[J]. Applied Intelligence, 2025, 55(2): 195. doi: 10.1007/s10489-024-05908-x

方法	AVEC2013		AVEC2014
方法	RMSE	MAE	RMSE	MAE
TSN	8.37	6.61	8.46	6.77
Spatial	8.89	7.02	9.31	7.47
Temporal	8.93	7.08	9.17	7.25
Sbranch	8.95	6.77	9.05	6.98
Lbranch	9.08	6.74	9.26	7.11
Features-level Fusion	8.54	6.86	8.84	6.86
MTDN	8.04	6.32	8.21	6.57

采样数量	AVEC2013		AVEC2014
采样数量	RMSE	MAE	RMSE	MAE
8	8.44	6.57	8.93	7.07
16	8.04	6.32	8.21	6.57
24	8.61	6.70	8.57	6.84

方法	RMSE	MAE
LPQ+SVR^[24]	13.61	10.88
MHH+PLS^[20]	11.19	9.14
LBP-TOP+SVR^[13]	8.91	7.08
DCNNs^[34]	9.82	7.58
RNN-C3D^[26]	9.28	7.37
Global-Local C3D^[37]	8.26	6.40
ResNet50^[35]	8.25	6.30
MSN^[17]	7.90	5.98
Bi-LSTM^[29]	8.93	7.04
LQGDNet^[16]	8.20	6.38
MDN-152^[19]	7.55	6.24
DAER^[23]	8.13	6.28
DMSN^[31]	7.66	6.14
MTDAN^[33]	8.08	6.14
STA-DRN^[38]	7.98	6.15
LSCAformer^[32]	7.69	5.89
LMTformer^[39]	7.75	6.12
Ours	8.04	6.32

方法	RMSE	MAE
LGBP-TOP+SVR^[25]	10.86	8.86
MHH+PLS^[20]	10.50	8.44
LBP-TOP+SVR^[13]	8.91	7.08
DCNNs^[34]	9.55	7.47
RNN-C3D^[26]	9.20	7.22
Global-Local C3D^[37]	8.31	6.59
ResNet50^[35]	8.23	6.15
MSN^[17]	7.61	5.82
Bi-LSTM^[29]	8.78	6.86
LQGDNet^[16]	7.84	6.08
MDN-152^[19]	7.65	6.06
DAER^[23]	8.07	6.14
DMSN^[31]	7.50	5.69
MTDAN^[33]	7.93	6.35
STA-DRN^[38]	7.75	6.00
LSCAformer^[32]	7.55	5.91
LMTformer^[39]	7.97	6.05
Ours	8.21	6.57

方法	AVEC2013		AVEC2014		Param/M	FLOPs/G
方法	RMSE	MAE	RMSE	MAE	Param/M	FLOPs/G
I3D	8.66	6.64	8.55	6.36	≈13	6.99
T3D	8.75	6.76	8.55	6.54	≈68	51.64
ResNet-50	8.25	6.30	8.23	6.15	≈25	3.80
MDN-50	8.13	6.39	8.16	6.45	≈21	7.40
MDN-152	7.55	6.24	7.65	6.06	≈52	13.36
Ours	8.04	6.32	8.21	6.57	1.03	1.97