Improving Adversarial Transferability on Vision-Language Pre-Training Models via Block Shuffle and Rotation

doi:10.11871/jfdc.issn.2096-742X.2025.02.013

Abstract

Abstract:

[Purpose] This study focuses on the vulnerability of Visual-Language Pretraining (VLP) models to adversarial examples. The aim is to propose a method to enhance the transferability of adversarial examples to address related security risks. [Literature Review] A summary and analysis of existing relevant studies have been conducted. [Application Background] Currently, VLP models are susceptible to adversarial examples, which pose significant security risks. Moreover, black-box transfer attacks are more reflective of real-world scenarios and thus worthy of more research compared to white-box adversarial attacks. [Methods] A transfer attack method based on block shuffle and rotation is proposed. When generating adversarial images and adversarial texts, operations based on block shuffle and rotation are added to increase the diversity of samples, thereby enhancing the adversarial transferability. [Results] Experiments on the Flickr30K dataset have verified the effectiveness of the proposed method. [Limitations] The adversarial transferability still needs to be further improved. [Conclusion] The proposed transfer attack method based on block shuffle and rotation can effectively improve the adversarial transferability of VLP models.

Key words: adversarial examples, adversarial transferability, vision-language pre-training model

WANG Wenbin,GAO Siyuan,GAO Manda,LIANG Ling,YANG Guangjun,HE Bangyan,LIU Yaozu. Improving Adversarial Transferability on Vision-Language Pre-Training Models via Block Shuffle and Rotation[J]. Frontiers of Data and Computing, 2025, 7(2): 130-140, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2025.02.013.

Figures/Tables 2

Table 1

Comparison between our method and baseline methods"

源模型	攻击方法	ALBEF		TCL		$C L I P V i T$		$C L I P C N N$
源模型	攻击方法	TR R@1	IR R@1	TR R@1	IR R@1	TR R@1	IR R@1	TR R@1	IR R@1
ALBEF	PGD	52.45*	58.65*	3.06	6.79	8.96	13.21	10.34	14.65
	BERT-Attack	11.57*	27.46*	12.64	28.07	29.33	43.17	32.69	46.11
	Sep-Attack	65.69*	73.95*	17.60	32.95	31.17	45.23	32.82	45.49
	Co-Attack	77.16*	83.86*	15.21	29.49	23.60	36.48	25.12	38.89
	SGA	97.50*	97.24*	45.84	55.24	32.39	44.14	36.02	46.76
	Ours	97.60*	97.75*	47.63	58.26	38.04	47.10	42.27	50.46
TCL	PGD	6.15	10.78	77.87*	79.48*	7.48	13.72	10.34	15.33
	BERT-Attack	11.89	26.82	14.54*	29.17*	29.69	44.49	33.46	46.07
	Sep-Attack	20.13	36.48	84.72*	86.07*	31.29	44.65	33.33	45.80
	Co-Attack	23.15	40.04	77.94*	85.59*	27.85	41.19	30.74	44.11
	SGA	49.53	60.20	98.42*	98.79*	34.23	45.14	37.16	48.16
	Ours	52.35	62.11	99.05*	99.14*	38.16	48.65	39.85	51.84
$C L I P V i T$	PGD	2.50	4.93	4.85	8.17	70.92*	78.61*	5.36	8.44
	BERT-Attack	9.59	22.64	11.80	25.07	28.34*	39.08*	30.40	37.43
	Sep-Attack	9.59	23.25	11.38	25.60	79.75*	86.79*	30.78	39.76
	Co-Attack	10.57	24.33	11.94	26.69	93.25*	95.86*	32.52	41.82
	SGA	12.72	27.25	14.44	29.83	99.02*	99.07*	39.72	47.38
	Ours	14.39	29.91	15.38	31.40	99.63*	99.19*	45.34	50.91
$C L I P C N N$	PGD	2.09	4.82	4.00	7.81	1.10	6.60	86.46*	92.25*
	BERT-Attack	8.86	23.27	12.33	25.48	27.12	37.44	30.40*	40.10*
	Sep-Attack	8.55	23.41	12.64	26.12	28.34	39.43	91.44*	95.44*
	Co-Attack	8.79	23.74	13.10	26.07	28.79	40.03	94.76*	96.89*
	SGA	11.16	25.07	14.44	27.40	31.04	41.78	99.36*	99.49*
	Ours	12.30	26.50	14.01	28.64	34.97	45.30	99.36*	99.59*

Table 1

Table 2

References 49

[1]	ZHANG J, YI Q, SANG J. Towards adversarial attack on vision-language pre-training models[C]// Proceedings of the ACM International Conference on Multimedia, 2022: 5005-5013.
[2]	LU D, WANG Z, WANG T, et al. Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models[C]// Proceedings of the IEEE International Conference on Computer Vision, 2023: 102-111.
[3]	REN S, HE K, GIRSHICK R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 39(6): 1137-1149.
[4]	JIANG H Z, MISRA I, ROHRBACH M, et al. In defense of grid features for visual question answering[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 10267-10276.
[5]	ZHOU L, PALANGI H, ZHANG L, et al. Unified vision-language pre-training for image captioning and vqa[C]// Proceedings of the AAAI conference on artificial intelligence, 2020, 34(7): 13041-13049.
[6]	DOU Z Y, XU Y, GAN Z, et al. An empirical study of training end-to-end vision-and-language transformers[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022: 18166-18176.
[7]	TAYLOR W L. “Cloze procedure”: A new tool for measuring readability[J]. Journalism Quarterly, 1953, 30(4): 415-433.
[8]	CHEN Y C, LI L, YU L, et al. Uniter: Universal image-text representation learning[C]// Proceedings of the European Conference on Computer Vision, 2020: 104-120.
[9]	LI J, SELVARAJU R, GOTMARE A, et al. Align before fuse: Vision and language representation learning with momentum distillation[J]. Advances in Neural Information Processing Systems, 2021, 34: 9694-9705.
[10]	WANG Z, YU J, YU A W, et al. Simvlm: Simple visual language model pretraining with weak supervision[J]. arXiv preprint arXiv: 2021, 2108.10904.
[11]	LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft coco: Common objects in context[C]// Proceedings of the European Conference on Computer Vision, 2014: 740-755.
[12]	KRISHNA R, ZHU Y, GROTH O, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations[J]. International Journal of Computer Vision, 2017, 123: 32-73.
[13]	SHARMA P, DING N, GOODMAN S, et al. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning[C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers), 2018: 2556-2565..
[14]	CHANGPINGYO S, SHARMA P, DING N, et al. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 3558-3568.
[15]	KAY W, CARREIRA J, SIMONYAN K, et al. The kinetics human action video dataset[J]. arXiv preprint arXiv: 2017, 1705.06950.
[16]	LU J, BATRA D, PARIKH D, et al. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[J]. Advances in Neural Information Processing Systems, 2019, 32.
[17]	LI X J, YIN X, LI C Y, et al. OSCAR: Object-semantics aligned pretraining for vision-language tasks[C]. In Proceedings of the 16th European Conference on Computer Vision, 2020: 121-137.
[18]	SZEGEDY C, ZAREMBA W, SUTSKEVER I, et al. Intriguing properties of neural networks[OL]// arXiv preprint arXiv, 2013: 1312.6199.
[19]	DONG Y, PANG T, SU H, et al. Evading defenses to transferable adversarial examples by translation-invariant attacks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019: 4312-4321.
[20]	LIN J, SONG C, HE K, et al. Nesterov accelerated gradient and scale invariance for adversarial attacks[J]. arXiv preprint arXiv, 2019: 1908.06281.
[21]	WU W B, SU Y X, LYU M R, et al. Improving the transferability of adversarial samples with adversarial transformations[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 9024-9033.
[22]	LI Y, BAI S, XIE C, et al. Regional Homogeneity: Towards Learning Transferable Universal Adversarial Perturbations Against Defenses[M]//Computer Vision-ECCV 2020, Lecture Notes in Computer Science, 2020: 795-813.
[23]	BYUN J, CHO S, KWON M J, et al. Improving the transferability of targeted adversarial examples through object-based diverse input[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022: 15244-15253.
[24]	WANG X, HE X, WANG J, et al. Admix: Enhancing the transferability of adversarial attacks[C]// Proceedings of the IEEE International Conference on Computer Vision, 2021: 16158-16167.
[25]	DONG Y, LIAO F, Pang T, et al. Boosting adversarial attacks with momentum[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 9185-9193.
[26]	XIONG Y, LIN J, ZHANG M, et al. Stochastic variance reduced ensemble adversarial attack for boosting the adversarial transferability[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022: 14983-14992.
[27]	ZHU H, REN Y, SUI X, et al. Boosting adversarial transferability via gradient relevance attack[C]// Proceedings of the IEEE International Conference on Computer Vision, 2023: 4741-4750.
[28]	LI M, DENG C, LI T, et al. Towards transferable targeted attack[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020: 641-649.
[29]	QIN Z, FAN Y, LIU Y, et al. Boosting the transferability of adversarial attacks with reverse adversarial perturbation[J]. Advances in Neural Information Processing Systems, 2022, 35: 29845-29858.
[30]	MA W, LI Y, JIA X, et al. Transferable adversarial attack for both vision transformers and convolutional networks via momentum integrated gradients[C]// Proceedings of the IEEE International Conference on Computer Vision, 2023: 4630-4639.
[31]	ZHANG C, BENZ P, KARJAUV A, et al. Investigating top-k white-box and transferable black-box attack[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022: 15085-15094.
[32]	XIAO Z, GAO X, FU C, et al. Improving transferability of adversarial patches on face recognition with generative models[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021: 11845-11854.
[33]	LI Q Z, GUO Y W, ZUO W M, et al. Making substitute models more bayesian can enhance transferability of adversarial examples[J]. arXiv preprint arXiv, 2023: 2302. 05086.
[34]	ZHAO Z, LIU Z, LARSON M. On success and simplicity: A second look at transferable targeted attacks[J]. Advances in Neural Information Processing Systems, 2021, 34: 6115-6128.
[35]	FANG S, LI J, LIN X, et al. Learning to learn transferable attack[C]// Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(1): 571-579.
[36]	XU X, ZHANG J Y, MA E, et al. Adversarially robust models may not transfer better: Sufficient conditions for domain transferability from the view of regularization[C]// International Conference on Machine Learning, 2022: 24770-24802.
[37]	QIAN Y, HE S, ZHAO C, et al. Lea2: A lightweight ensemble adversarial attack via non-overlapping vulnerable frequency regions[C]// Proceedings of the IEEE International Conference on Computer Vision, 2023: 4510-4521.
[38]	CHEN B, YIN J, CHEN S, et al. An adaptive model ensemble adversarial attack for boosting adversarial transferability[C]// Proceedings of the IEEE International Conference on Computer Vision, 2023: 4489-4498.
[39]	HUANG Q, KATSMAN I, He H, et al. Enhancing adversarial example transferability with an intermediate level attack[C]// Proceedings of the IEEE International Conference on Computer Vision, 2019: 4733-4742.
[40]	GUO Y, LI Q, CHEN H. Backpropagating linearly improves transferability of adversarial examples[J]. Advances in Neural Information Processing Systems, 2020, 33: 85-95.
[41]	GUBRI M, CORDY M, PAPADAKIS M, et al. Lgv: Boosting adversarial example transferability from large geometric vicinity[C]// Proceedings of the European Conference on Computer Vision, 2022: 603-618.
[42]	POURSAEED O, Katsman I, Gao B, et al. Generative adversarial perturbations[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 4422-4431.
[43]	NASSER M, KHAN S, HAYAT M, et al. On generating transferable targeted perturbations[C]// Proceedings of the IEEE International Conference on Computer Vision, 2021: 7708-7717.
[44]	KIM W J, HONG S, YOON S E. Diverse generative perturbations on attention space for transferable adversarial attacks[C]// IEEE International Conference on Image Processing, 2022: 281-285.
[45]	FENG Y, WU B, FAN Y, et al. Boosting black-box attack with partially transferred conditional adversarial distribution[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022: 15095-15104.
[46]	ZHAO A, CHU T, LIU Y, et al. Minimizing maximum model discrepancy for transferable black-box targeted attacks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023: 8153-8162.
[47]	YANG X, DONG Y, PANG T, et al. Boosting transferability of targeted adversarial examples via hierarchical generative networks[C]// Proceedings of the European Conference on Computer Vision, 2022: 725-742.
[48]	PLUMMER B A, WANG L, CERVANTES C M, et al. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models[C]// Proceedings of the IEEE International Conference on Computer Vision, 2015: 2641-2649.
[49]	WANG K, HE X, WANG W, et al. Boosting adversarial transferability by block shuffle and rotation[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024: 24336-24346.

Attack	TR R@1	TR R@5	TR R@10	IR R@1	IR R@5	IR R@10
ImgAttack	39.59	18.39	11.23	47.99	30.52	24.51
TxtAttack	40.61	18.6	12.46	50.94	33.24	26.74
ImgAttack+TxtAttack	42.27	19.24	11.74	50.46	33.31	26.56