基于生成对抗网络的有遮挡人脸修复方法

doi:10.11871/jfdc.issn.2096-742X.2022.04.012

数据与计算发展前沿 ›› 2022, Vol. 4 ›› Issue (4): 123-131.

CSTR: 32002.14.jfdc.CN10-1649/TP.2022.04.012

doi: 10.11871/jfdc.issn.2096-742X.2022.04.012

基于生成对抗网络的有遮挡人脸修复方法

石雪梅¹(),朱克亮¹,张祥民¹,张树涛^2,³,陈良锋^2,^*()

1.国网安徽省电力有限公司建设分公司,安徽合肥 230022
2.中国科学院合肥物质科学研究院,安徽合肥 230026
3.中国科学技术大学,安徽合肥 230031

收稿日期:2021-07-23 出版日期:2022-08-20 发布日期:2022-08-10
通讯作者: 陈良锋
作者简介:石雪梅,国网安徽省电力有限公司建设分公司,硕士,研究方向为计算机技术。
本文中负责撰稿。
SHI Xuemei, master's degree, works in State Grid Anhui Electric Power Co., Ltd, Construction Branch. Her current research direction is computer technology.
In this paper, she is responsible for the paper writing.
E-mail: 524104473@qq.com|陈良锋,中国科学院合肥物质科学研究院,博士,工程师,研究方向为机器视觉。
在本文中负责核心算法论证。
CHEN Liangfeng, Ph.D., is an engineer of Hefei Institutes of Physical Science, Chinese Academy of Sciences. His cur-rent research interests include machine visi-on.
In this paper, he is responsible for core algorithm demonstration.
E-mail: quinear@hfcas.ac.cn
基金资助:
国家电网公司科技项目(B3120A190005)

Occluded Face Inpainting Method Based on Generative Adversarial Networks

SHI Xuemei¹(),ZHU Keliang¹,ZHANG Xiangmin¹,ZHANG Shutao^2,³,CHEN Liangfeng^2,^*()

1. State Grid Anhui Electric Power Co, Ltd. Construction Company, Hefei, Anhui 230022, China
2. Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, Anhui 230026, China
3. University of Science and Technology of China, Hefei, Anhui 230031, China

Received:2021-07-23 Online:2022-08-20 Published:2022-08-10
Contact: CHEN Liangfeng

摘要/Abstract

摘要：

【目的】实现有遮挡人脸图像修复,以提升人脸识别系统中有遮挡条件下人脸识别的准确率。【应用背景】在无感人脸识别场景中,人脸图像常因帽子、口罩等遮挡物影响,现有人脸识别算法出现准确率下降甚至无法识别现象。【方法】本文综合考虑图像内容相似性、真实性和生成的图像连贯性,提出一种由单一生成器和双判别器组成的生成对抗网络,将生成器生成的人脸被遮挡部分输入局部判别器,并将其与去除遮挡部分的原始人脸图像融合后输入全局判别器,提升生成器的稳定性及精确性。【结果】采用LFW人脸数据集实验分析可知,本文方法更好地实现了有遮挡的人脸图像修复,在图像结构相似度(SSIM)和峰值信噪比(PSNR)上,较经典DCGAN提高了67%和40%。【结论】针对实际应用场景,以图像内容相似性为生成对抗网络输入条件,提高了网络的收敛效率和稳定性,可进一步优化网络结构、降低层间损失以提升生成图像的质量。

关键词: 人脸修复, 生成对抗网络, 图像融合, 卷积神经网络, 损失函数

Abstract:

[Objective] Face recognition, a basic technology in artificial intelligence applications, would be invalid due to the occlusion phenomenon. To improve the face recognition accuracy in the presence of partial occlusions, we propose a method to generate occluded regions of a face image automatically. [Context] The accuracy is the essential factor in face recognition applications. But its performance drastically decreases, and even to zero, when face images are occluded by objects such as helmets, glasses, and masks. [Methods] Taking image attributes into accounts, such as similarity, validity, and consistency, we present an improved generative adversarial network which is composed of one generator and two discriminators. To improve the stability and accuracy of the generator, the occluded part of the face generated by the generator is input to the local discriminator, and together with the original face image without the occluded part serve as the input to the globe discriminator. [Results] The experiments conducted on the LFW dataset show that the Structural Similarity (SSIM) of our method is 67% higher than DCGAN, and the Peak Signal to Noise Ratio (PSNR) is 40% higher. [Conclusions] The results show that prior conditions such as the similarity between pair images can improve the stability and convergence rate of a generator. In future work, a better network architecture that reduces interlayer loss would be proposed to improve the quality of generated images.

Key words: face inpainting, generative adversarial network, image fusion, convolutional neural network, loss function

石雪梅,朱克亮,张祥民,张树涛,陈良锋. 基于生成对抗网络的有遮挡人脸修复方法[J]. 数据与计算发展前沿, 2022, 4(4): 123-131.

SHI Xuemei,ZHU Keliang,ZHANG Xiangmin,ZHANG Shutao,CHEN Liangfeng. Occluded Face Inpainting Method Based on Generative Adversarial Networks[J]. Frontiers of Data and Computing, 2022, 4(4): 123-131, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2022.04.012.

图/表 11

图1

图2

图3

图4

表1

表2

表3

表4

图5

图6

表5

参考文献 22

[1]	赵露露, 沈玲, 洪日昌. 图像修复研究进展综述[J]. 计算机科学, 2021, 48(03):14-26.
[2]	强振平, 何丽波, 陈旭, 等. 深度学习图像修复方法综述[J]. 中国图象图形学报, 2019, 24(03):447-463.
[3]	宋文净, 金晅宏. 基于生成对抗网络的面部遮挡还原算法研究[J]. 电子测量技术, 2020, 43(23):109-114.
[4]	SHEN J, CHAN T F. Mathematical models for local non-texture inpaintings[J]. SIAM Journal on Applied Mathe-matics, 2002, 62(3):1019-1043.
[5]	BARNES C, SHECHTMAN E, FINKELSTEIN A, et al. PatchMatch: A randomized correspondence algorithm for structural image editing[J]. ACM Transactions on Graphics, 2009, 28(3):21-28.
[6]	Goodfellow I J, Pouget-Abadie J, Mirza M, et al. Gen-erative adversarial nets[C]. Proceedings of the 27th Inter-national Conference on Neural Information Processing Systems.MIT Press, 2014, 2: 2672–2680.
[7]	Wang Y, Tao X, Qi X J, et al. Image inpainting via gen-erative multi-column convolutional neural networks[C]. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2018: 329-338.
[8]	刘昱, 刘厚泉. 基于对抗训练和卷积神经网络的面部图像修复[J]. 计算机工程与应用, 2019, 55(02): 110-115+136.
[9]	孙全, 曾晓勤. 基于生成对抗网络的图像修复[J]. 计算机科学, 2018, 45(12):229-234+261.
[10]	武文杰, 王红蕾. 基于生成对抗网络的遮挡人脸图像修复的改进与实现[J]. 计算机应用与软件, 2021, 38(01):217-221+249.
[11]	Han C, Wang J L. Face Image Inpainting With Evo-lutionary Generators[J]. IEEE Signal Processing Letters, 2021, 28:190-193. doi: 10.1109/LSP.2020.3048608
[12]	YEH R, CHEN C, LIM T Y, et al. Semantic image inpa-inting with perceptual and contextual losses[C]. In Proce-edings of IEEE Conference on Computer Vision and Pattern Recognition, 2017: 6882-6890.
[13]	Yu L, Zhu D Q, He J. Semantic segmentation guided face inpainting based on SN-PatchGAN[C]. 2020 13th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), 2020:110-115.
[14]	Zhao Y, Liu L, Liu H, et al. Face Inpainting based on Improved WGAN-modified[C]. 2020 International Sym-posium on Autonomous Systems (ISAS), 2020:7-11.
[15]	Zhang H, Li T. Semantic Face Image Inpainting based on Generative Adversarial Network[C]. 2020 35th Youth Academic Annual Conference of Chinese Association of Automation (YAC), 2020:530-535.
[16]	Iizuka Satoshi, Simo-Serra Edgar, Ishikawa Hiroshi. Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4):4-5.
[17]	刘波宁, 翟东海. 基于双鉴别网络的生成对抗网络图像修复方法[J]. 计算机应用, 2018, 38(12): 3557-3562+3595. doi: 10.11772/j.issn.1001-9081.2018051097
[18]	李天成, 何嘉. 一种基于生成对抗网络的图像修复算法[J]. 计算机应用与软件, 2019, 36(12): 195-200+ 267.
[19]	Zhang C, Wang T. Image Inpainting Using Double Dis-criminator Generative Adversarial Networks[C]. 2^nd International Seminar on Computer Science and Engin-eering Technology (SCSET) 2020, 2020:1-6.
[20]	HUANG G B, RAMESH M, BERG T, et al. Labeled faces in the wild: A database for studying face recognition in unconstrained environments[C]. In Proceedings of IEEE International Workshop on Performance Evalua-tion for Tracking and Surveillance, 2008:7-49.
[21]	WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: From error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612. doi: 10.1109/TIP.2003.819861
[22]	AVCIBAS I, SANBUR B, SAYOOD K. Statistical evalu-ation of image quality measures[J]. Journal of Electronic Imaging, 2002, 11(2): 206-213. doi: 10.1117/1.1455011

类型	输入	卷积核	步长	卷积核数	输出
Conv, lReLU	96×96×3	5×5	2	64	48×48×64
Conv, lReLU	48×48×64	5×5	2	128	24×24×128
Conv, lReLU	24×24×128	5×5	2	256	12×12×256
Conv, lReLU	12×12×256	5×5	2	512	6×6×512
Deconv, ReLU	6×6×512	5×5	2	256	12×12×256
Deconv, ReLU	12×12×256	5×5	2	128	24×24×128
Deconv, ReLU	24×24×128	5×5	2	64	48×48×64
Deconv, tanh	48×48×64	5×5	2	3	96×96×3

类型	输入	卷积核	步长	卷积核数	输出
Conv, lReLU	96×96×3	5×5	2	64	48×48×64
Conv, BN, lReLU	48×48×64	5×5	2	128	24×24×128
Conv, BN, lReLU	24×24×128	5×5	2	256	12×12×256
Conv, BN, lReLU	12×12×256	5×5	2	512	6×6×512
Linear, sigmoid	6×6×512	—	—	—	1

类型	输入	卷积核	步长	卷积核数	输出
Conv, lReLU	20×40×3	5×5	2	64	10×20×64
Conv, BN, lReLU	10×20×64	5×5	2	128	5×10×128
Conv, BN, lReLU	5×10×128	5×5	2	256	3×5×256
Conv, BN, lReLU	3×5×256	5×5	2	512	2×3×512
Linear, sigmoid	2×3×512	—	—	—	1

方法	PSNR	SSIM
DCGAN	20.57	0.5030
无局部判别器	20.88	0.6282
α=0.8	23.46	0.8051
α=0.7	24.78	0.8045
α=0.6	26.41	0.8390
α=0.5	28.90	0.8103

基于生成对抗网络的有遮挡人脸修复方法

Occluded Face Inpainting Method Based on Generative Adversarial Networks

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 22

相关文章 5

编辑推荐

Metrics

本文评价

[1]	肖楠,周明珠,邢军,罗泽,李晓辉. 基于高分辨率网络和注意力机制的真伪卷烟包装鉴别[J]. 数据与计算发展前沿, 2021, 3(5): 118-129.
[2]	张猛,李健. 鸟类音频数据预处理方法[J]. 数据与计算发展前沿, 2021, 3(5): 130-140.
[3]	陈子健,李俊,岳兆娟,赵泽方. 基于自编码器与属性信息的混合推荐模型[J]. 数据与计算发展前沿, 2021, 3(3): 148-155.
[4]	刘晓东,倪浩然. 深度学习技术在学科融合研究中的应用[J]. 数据与计算发展前沿, 2020, 2(5): 99-109.
[5]	欧阳与点,谢鲲. 网络性能数据恢复算法[J]. 数据与计算发展前沿, 2020, 2(3): 55-65.

输入：有遮挡的人脸图像
输出：无遮挡的人脸图像
1. 图像预处理,将人脸数据集中的图像裁剪为相同像素大小 $h\times w\times 3$,对像素值的取值范围进行归一化
2. for i in epoch: //epoch为迭代次数
3. for j in dataset_size/batch_size: //datas_size为数据集大小,batch_size为批大小
4. AdamOptimizer(learning_rate, beta1=beta). minimize ( ${{L}_{D}}$)//采用Adam优化器最小化判别器的损失函数 ${{L}_{D}}$,learning_rate为学习率,beta为衰减率
5. AdamOptimizer(learning_rate, beta1=beta). minimize ( ${{L}_{G}}$)//采用Adam优化器最小化生成器的损失函数 ${{L}_{G}}$,learning_rate为学习率,beta为衰减率
6. end for
7. end for