通过块打乱和旋转提升对视觉-语言模型的对抗迁移性

doi:10.11871/jfdc.issn.2096-742X.2025.02.013

数据与计算发展前沿 ›› 2025, Vol. 7 ›› Issue (2): 130-140.

CSTR: 32002.14.jfdc.CN10-1649/TP.2025.02.013

doi: 10.11871/jfdc.issn.2096-742X.2025.02.013

通过块打乱和旋转提升对视觉-语言模型的对抗迁移性

王文彬¹(),高思远^1,^*(),高满达¹,梁凌¹,杨光俊¹,何邦彦²,刘耀祖²

1.国家能源集团新能源技术研究院有限公司，北京 102209
2.中国科学院自动化研究所，北京 100190

收稿日期:2024-12-09 出版日期:2025-04-20 发布日期:2025-04-23
通讯作者: 高思远
作者简介:王文彬，国家能源集团新能源技术研究院有限公司，基石运营技术研究中心主任，主要从事人工智能技术在能源行业的应用研究工作。
本文承担工作为方法的提出和实现。
WANG Wenbin, is the director of the Cornerstone Operation Technology Research Center, CHN Energy New Energy Technology Research Institute Co., Ltd. He is mainly engaged in research on the application of artificial intelligence technology in the energy industry.
In this paper, he is mainly responsible for method proposal and method algorithm realization.
E-mail: 16080112@ceic.com|高思远，国家能源集团新能源技术研究院有限公司，基石运营技术研究中心工程师，博士，主要研究方向为深度学习，模型安全等。
本文承担工作为工作的整体协调与推进。
GAO Siyuan, Ph.D., is an engineer at the Cornerstone Operation Technology Research Center, CHN Energy New Energy Technology Research Institute Co., Ltd. Her research interests include deep learning, model security, and related areas.
In this paper, she is mainly responsible for overall coordination and advancement of work.
E-mail: 20065237@ceic.com
基金资助:
国家能源集团科技创新项目“火电厂人工智能运营体系典型应用场景样本库模型库研究”(GJNY-23-99)

Improving Adversarial Transferability on Vision-Language Pre-Training Models via Block Shuffle and Rotation

WANG Wenbin¹(),GAO Siyuan^1,^*(),GAO Manda¹,LIANG Ling¹,YANG Guangjun¹,HE Bangyan²,LIU Yaozu²

1. CHN Energy New Energy Technology Research Institute Co., Ltd, Beijing 102209, China
2. Institute of Automation, Chinese Acadamy of Science, Beijing 100190, China

Received:2024-12-09 Online:2025-04-20 Published:2025-04-23
Contact: GAO Siyuan

摘要/Abstract

摘要：

【目的】研究视觉-语言预训练VLP模型易受对抗样本攻击的问题，旨在提出一种能提高对抗样本迁移性的方法以应对相关安全风险。【文献范围】对现有相关研究进行了总结与分析。【应用背景】当前VLP模型易受对抗样本攻击，其带来重大安全风险，且黑盒迁移攻击相比白盒对抗攻击更能反映现实场景，更具研究意义。【方法】提出了基于块打乱和旋转的迁移攻击方法，在生成对抗图像和对抗文本时，加入基于块打乱和旋转操作，以此提升样本的多样性，从而提升对抗迁移性。【结果】在Flickr30K数据集上进行的实验，验证了所提方法的有效性。【局限】对抗迁移性仍有待进一步提升。【结论】所提出的基于块打乱和旋转的迁移攻击方法，能够有效提高对VLP模型的对抗迁移性。

关键词: 对抗样本, 对抗迁移性, 视觉-语言预训练模型

Abstract:

[Purpose] This study focuses on the vulnerability of Visual-Language Pretraining (VLP) models to adversarial examples. The aim is to propose a method to enhance the transferability of adversarial examples to address related security risks. [Literature Review] A summary and analysis of existing relevant studies have been conducted. [Application Background] Currently, VLP models are susceptible to adversarial examples, which pose significant security risks. Moreover, black-box transfer attacks are more reflective of real-world scenarios and thus worthy of more research compared to white-box adversarial attacks. [Methods] A transfer attack method based on block shuffle and rotation is proposed. When generating adversarial images and adversarial texts, operations based on block shuffle and rotation are added to increase the diversity of samples, thereby enhancing the adversarial transferability. [Results] Experiments on the Flickr30K dataset have verified the effectiveness of the proposed method. [Limitations] The adversarial transferability still needs to be further improved. [Conclusion] The proposed transfer attack method based on block shuffle and rotation can effectively improve the adversarial transferability of VLP models.

Key words: adversarial examples, adversarial transferability, vision-language pre-training model

王文彬,高思远,高满达,梁凌,杨光俊,何邦彦,刘耀祖. 通过块打乱和旋转提升对视觉-语言模型的对抗迁移性[J]. 数据与计算发展前沿, 2025, 7(2): 130-140.

WANG Wenbin,GAO Siyuan,GAO Manda,LIANG Ling,YANG Guangjun,HE Bangyan,LIU Yaozu. Improving Adversarial Transferability on Vision-Language Pre-Training Models via Block Shuffle and Rotation[J]. Frontiers of Data and Computing, 2025, 7(2): 130-140, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2025.02.013.

图/表 2

表1

本文方法与基线方法的比较"

源模型	攻击方法	ALBEF		TCL		$C L I P V i T$		$C L I P C N N$
源模型	攻击方法	TR R@1	IR R@1	TR R@1	IR R@1	TR R@1	IR R@1	TR R@1	IR R@1
ALBEF	PGD	52.45*	58.65*	3.06	6.79	8.96	13.21	10.34	14.65
	BERT-Attack	11.57*	27.46*	12.64	28.07	29.33	43.17	32.69	46.11
	Sep-Attack	65.69*	73.95*	17.60	32.95	31.17	45.23	32.82	45.49
	Co-Attack	77.16*	83.86*	15.21	29.49	23.60	36.48	25.12	38.89
	SGA	97.50*	97.24*	45.84	55.24	32.39	44.14	36.02	46.76
	Ours	97.60*	97.75*	47.63	58.26	38.04	47.10	42.27	50.46
TCL	PGD	6.15	10.78	77.87*	79.48*	7.48	13.72	10.34	15.33
	BERT-Attack	11.89	26.82	14.54*	29.17*	29.69	44.49	33.46	46.07
	Sep-Attack	20.13	36.48	84.72*	86.07*	31.29	44.65	33.33	45.80
	Co-Attack	23.15	40.04	77.94*	85.59*	27.85	41.19	30.74	44.11
	SGA	49.53	60.20	98.42*	98.79*	34.23	45.14	37.16	48.16
	Ours	52.35	62.11	99.05*	99.14*	38.16	48.65	39.85	51.84
$C L I P V i T$	PGD	2.50	4.93	4.85	8.17	70.92*	78.61*	5.36	8.44
	BERT-Attack	9.59	22.64	11.80	25.07	28.34*	39.08*	30.40	37.43
	Sep-Attack	9.59	23.25	11.38	25.60	79.75*	86.79*	30.78	39.76
	Co-Attack	10.57	24.33	11.94	26.69	93.25*	95.86*	32.52	41.82
	SGA	12.72	27.25	14.44	29.83	99.02*	99.07*	39.72	47.38
	Ours	14.39	29.91	15.38	31.40	99.63*	99.19*	45.34	50.91
$C L I P C N N$	PGD	2.09	4.82	4.00	7.81	1.10	6.60	86.46*	92.25*
	BERT-Attack	8.86	23.27	12.33	25.48	27.12	37.44	30.40*	40.10*
	Sep-Attack	8.55	23.41	12.64	26.12	28.34	39.43	91.44*	95.44*
	Co-Attack	8.79	23.74	13.10	26.07	28.79	40.03	94.76*	96.89*
	SGA	11.16	25.07	14.44	27.40	31.04	41.78	99.36*	99.49*
	Ours	12.30	26.50	14.01	28.64	34.97	45.30	99.36*	99.59*

表1

表2

参考文献 49

[1]	ZHANG J, YI Q, SANG J. Towards adversarial attack on vision-language pre-training models[C]// Proceedings of the ACM International Conference on Multimedia, 2022: 5005-5013.
[2]	LU D, WANG Z, WANG T, et al. Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models[C]// Proceedings of the IEEE International Conference on Computer Vision, 2023: 102-111.
[3]	REN S, HE K, GIRSHICK R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 39(6): 1137-1149.
[4]	JIANG H Z, MISRA I, ROHRBACH M, et al. In defense of grid features for visual question answering[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 10267-10276.
[5]	ZHOU L, PALANGI H, ZHANG L, et al. Unified vision-language pre-training for image captioning and vqa[C]// Proceedings of the AAAI conference on artificial intelligence, 2020, 34(7): 13041-13049.
[6]	DOU Z Y, XU Y, GAN Z, et al. An empirical study of training end-to-end vision-and-language transformers[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022: 18166-18176.
[7]	TAYLOR W L. “Cloze procedure”: A new tool for measuring readability[J]. Journalism Quarterly, 1953, 30(4): 415-433.
[8]	CHEN Y C, LI L, YU L, et al. Uniter: Universal image-text representation learning[C]// Proceedings of the European Conference on Computer Vision, 2020: 104-120.
[9]	LI J, SELVARAJU R, GOTMARE A, et al. Align before fuse: Vision and language representation learning with momentum distillation[J]. Advances in Neural Information Processing Systems, 2021, 34: 9694-9705.
[10]	WANG Z, YU J, YU A W, et al. Simvlm: Simple visual language model pretraining with weak supervision[J]. arXiv preprint arXiv: 2021, 2108.10904.
[11]	LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft coco: Common objects in context[C]// Proceedings of the European Conference on Computer Vision, 2014: 740-755.
[12]	KRISHNA R, ZHU Y, GROTH O, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations[J]. International Journal of Computer Vision, 2017, 123: 32-73.
[13]	SHARMA P, DING N, GOODMAN S, et al. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning[C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers), 2018: 2556-2565..
[14]	CHANGPINGYO S, SHARMA P, DING N, et al. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 3558-3568.
[15]	KAY W, CARREIRA J, SIMONYAN K, et al. The kinetics human action video dataset[J]. arXiv preprint arXiv: 2017, 1705.06950.
[16]	LU J, BATRA D, PARIKH D, et al. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[J]. Advances in Neural Information Processing Systems, 2019, 32.
[17]	LI X J, YIN X, LI C Y, et al. OSCAR: Object-semantics aligned pretraining for vision-language tasks[C]. In Proceedings of the 16th European Conference on Computer Vision, 2020: 121-137.
[18]	SZEGEDY C, ZAREMBA W, SUTSKEVER I, et al. Intriguing properties of neural networks[OL]// arXiv preprint arXiv, 2013: 1312.6199.
[19]	DONG Y, PANG T, SU H, et al. Evading defenses to transferable adversarial examples by translation-invariant attacks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019: 4312-4321.
[20]	LIN J, SONG C, HE K, et al. Nesterov accelerated gradient and scale invariance for adversarial attacks[J]. arXiv preprint arXiv, 2019: 1908.06281.
[21]	WU W B, SU Y X, LYU M R, et al. Improving the transferability of adversarial samples with adversarial transformations[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 9024-9033.
[22]	LI Y, BAI S, XIE C, et al. Regional Homogeneity: Towards Learning Transferable Universal Adversarial Perturbations Against Defenses[M]//Computer Vision-ECCV 2020, Lecture Notes in Computer Science, 2020: 795-813.
[23]	BYUN J, CHO S, KWON M J, et al. Improving the transferability of targeted adversarial examples through object-based diverse input[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022: 15244-15253.
[24]	WANG X, HE X, WANG J, et al. Admix: Enhancing the transferability of adversarial attacks[C]// Proceedings of the IEEE International Conference on Computer Vision, 2021: 16158-16167.
[25]	DONG Y, LIAO F, Pang T, et al. Boosting adversarial attacks with momentum[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 9185-9193.
[26]	XIONG Y, LIN J, ZHANG M, et al. Stochastic variance reduced ensemble adversarial attack for boosting the adversarial transferability[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022: 14983-14992.
[27]	ZHU H, REN Y, SUI X, et al. Boosting adversarial transferability via gradient relevance attack[C]// Proceedings of the IEEE International Conference on Computer Vision, 2023: 4741-4750.
[28]	LI M, DENG C, LI T, et al. Towards transferable targeted attack[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020: 641-649.
[29]	QIN Z, FAN Y, LIU Y, et al. Boosting the transferability of adversarial attacks with reverse adversarial perturbation[J]. Advances in Neural Information Processing Systems, 2022, 35: 29845-29858.
[30]	MA W, LI Y, JIA X, et al. Transferable adversarial attack for both vision transformers and convolutional networks via momentum integrated gradients[C]// Proceedings of the IEEE International Conference on Computer Vision, 2023: 4630-4639.
[31]	ZHANG C, BENZ P, KARJAUV A, et al. Investigating top-k white-box and transferable black-box attack[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022: 15085-15094.
[32]	XIAO Z, GAO X, FU C, et al. Improving transferability of adversarial patches on face recognition with generative models[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021: 11845-11854.
[33]	LI Q Z, GUO Y W, ZUO W M, et al. Making substitute models more bayesian can enhance transferability of adversarial examples[J]. arXiv preprint arXiv, 2023: 2302. 05086.
[34]	ZHAO Z, LIU Z, LARSON M. On success and simplicity: A second look at transferable targeted attacks[J]. Advances in Neural Information Processing Systems, 2021, 34: 6115-6128.
[35]	FANG S, LI J, LIN X, et al. Learning to learn transferable attack[C]// Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(1): 571-579.
[36]	XU X, ZHANG J Y, MA E, et al. Adversarially robust models may not transfer better: Sufficient conditions for domain transferability from the view of regularization[C]// International Conference on Machine Learning, 2022: 24770-24802.
[37]	QIAN Y, HE S, ZHAO C, et al. Lea2: A lightweight ensemble adversarial attack via non-overlapping vulnerable frequency regions[C]// Proceedings of the IEEE International Conference on Computer Vision, 2023: 4510-4521.
[38]	CHEN B, YIN J, CHEN S, et al. An adaptive model ensemble adversarial attack for boosting adversarial transferability[C]// Proceedings of the IEEE International Conference on Computer Vision, 2023: 4489-4498.
[39]	HUANG Q, KATSMAN I, He H, et al. Enhancing adversarial example transferability with an intermediate level attack[C]// Proceedings of the IEEE International Conference on Computer Vision, 2019: 4733-4742.
[40]	GUO Y, LI Q, CHEN H. Backpropagating linearly improves transferability of adversarial examples[J]. Advances in Neural Information Processing Systems, 2020, 33: 85-95.
[41]	GUBRI M, CORDY M, PAPADAKIS M, et al. Lgv: Boosting adversarial example transferability from large geometric vicinity[C]// Proceedings of the European Conference on Computer Vision, 2022: 603-618.
[42]	POURSAEED O, Katsman I, Gao B, et al. Generative adversarial perturbations[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 4422-4431.
[43]	NASSER M, KHAN S, HAYAT M, et al. On generating transferable targeted perturbations[C]// Proceedings of the IEEE International Conference on Computer Vision, 2021: 7708-7717.
[44]	KIM W J, HONG S, YOON S E. Diverse generative perturbations on attention space for transferable adversarial attacks[C]// IEEE International Conference on Image Processing, 2022: 281-285.
[45]	FENG Y, WU B, FAN Y, et al. Boosting black-box attack with partially transferred conditional adversarial distribution[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022: 15095-15104.
[46]	ZHAO A, CHU T, LIU Y, et al. Minimizing maximum model discrepancy for transferable black-box targeted attacks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023: 8153-8162.
[47]	YANG X, DONG Y, PANG T, et al. Boosting transferability of targeted adversarial examples via hierarchical generative networks[C]// Proceedings of the European Conference on Computer Vision, 2022: 725-742.
[48]	PLUMMER B A, WANG L, CERVANTES C M, et al. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models[C]// Proceedings of the IEEE International Conference on Computer Vision, 2015: 2641-2649.
[49]	WANG K, HE X, WANG W, et al. Boosting adversarial transferability by block shuffle and rotation[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024: 24336-24346.

Attack	TR R@1	TR R@5	TR R@10	IR R@1	IR R@5	IR R@10
ImgAttack	39.59	18.39	11.23	47.99	30.52	24.51
TxtAttack	40.61	18.6	12.46	50.94	33.24	26.74
ImgAttack+TxtAttack	42.27	19.24	11.74	50.46	33.31	26.56

通过块打乱和旋转提升对视觉-语言模型的对抗迁移性

Improving Adversarial Transferability on Vision-Language Pre-Training Models via Block Shuffle and Rotation

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 2

参考文献 49

相关文章 1

编辑推荐

Metrics

本文评价