基于多尺度和循环生成对抗的连接式去雨网络

doi:10.11871/jfdc.issn.2096-742X.2023.05.011

数据与计算发展前沿 ›› 2023, Vol. 5 ›› Issue (5): 128-139.

CSTR: 32002.14.jfdc.CN10-1649/TP.2023.05.011

doi: 10.11871/jfdc.issn.2096-742X.2023.05.011

基于多尺度和循环生成对抗的连接式去雨网络

郎晓奇(),张娟^*()

上海工程技术大学，电子电气工程学院，上海 201620

收稿日期:2022-04-02 出版日期:2023-10-20 发布日期:2023-10-31
通讯作者: *张娟（E-mail: zhang-j@foxmail.com）
作者简介:郎晓奇，上海工程技术大学，电子电气工程学院，硕士研究生，主要研究方向为计算机视觉、图像去雾、图像去雨。
本文中负责提出研究思路，对方法进行设计实验，论文撰写。
LANG Xiaoqi is a postgraduate student at the School of Electrical and Electronic Engineering, Shanghai University of Engineering and Technology. His main research interests are computer vision, image dehazing, and image deraining.
In this paper, he is responsible for proposing research ideas, conducting experiments on the designed model, and writing the paper.
E-mail: m020219338@sues.edu.cn|张娟，上海工程技术大学，电子电气工程学院，副教授，硕士生导师，主要研究方向为图像处理、机器学习和计算机视觉。
本文中负责设计研究方案和框架，论文最终版本修订。
ZHANG Juan is currently an associate professor in the School of Electronic and Electrical Engineering, Shanghai University of Engineering Science. Her main research interests are image processing, machine learning, and computer vision.
In this paper, she is responsible for the design of the research protocol and framework and the revision of the final version of the paper.
E-mail: zhang-j@foxmail.com
基金资助:
国家自然科学基金(61801288)

Connected Deraining Network Based on Multi-Scale and Cyclic Generative Adversarial

LANG Xiaoqi(),ZHANG Juan^*()

School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai 201620, China

Received:2022-04-02 Online:2023-10-20 Published:2023-10-31

摘要/Abstract

摘要：

【目的】 图像去雨能够作为其他计算机视觉任务的预处理步骤，使自动驾驶、目标识别等其他计算机视觉任务的结果进一步得到提升。【方法】 本文将多尺度信息交换与循环生成对抗网络进行了连接，提出的方法根据训练步骤分为两个部分，首先通过多尺度信息交换得到雨水条纹信息进行初去雨，然后通过循环生成对抗网络对初去雨图像进行进一步增强，以得到效果最佳的去雨图像。【结果】 该方法能够有效地去除图像中的雨水信息，恢复出清晰的图像。本文去雨结果在PSNR（Peak Signal to Noise Ratio）和SSIM（Structural Similarity）评价指标上取得了较高的结果，能够更好地保留图像的细节。【结论】 通过在合成数据集以及真实图像上与其他图像去雨方法的结果进行对比，本文的方法取得了较好的效果，能够更好地为其他计算机视觉任务提供支持。

关键词: 图像处理, 图像去雨, 多尺度信息交换, 循环生成对抗网络

Abstract:

[Objective] Image deraining can be used as a preprocessing step for computer vision tasks so that the results of computer vision tasks such as automatic drive and target recognition can be improved. [Methods] In this paper, multi-scale information exchange is connected with a cyclic generative adversarial network. The proposed method is divided into two parts according to the training steps. First, the rain streak information is obtained through multi-scale information exchange for initial rain removal. Then the image with initial rain removal is further enhanced by a cyclic generative adversarial network so as to obtain the best rain removal effect. [Results] This method can effectively remove the rain information in the image and restore a clear image. This method presented in this paper has achieved good rain removal results in PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity) evaluation indexes and can better preserve the details of the image. [Conclusions] By comparing the results on synthetic datasets and real images with other image rain removal methods, this method has achieved better results and can provide better support for other computer vision tasks.

Key words: image processing, image deraining, multi-scale information exchange, cycle generative adversarial network

郎晓奇, 张娟. 基于多尺度和循环生成对抗的连接式去雨网络[J]. 数据与计算发展前沿, 2023, 5(5): 128-139.

LANG Xiaoqi, ZHANG Juan. Connected Deraining Network Based on Multi-Scale and Cyclic Generative Adversarial[J]. Frontiers of Data and Computing, 2023, 5(5): 128-139, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2023.05.011.

图/表 7

图1

表1

表2

表3

表4

图2

图3

参考文献 45

[1]	REN S, HE K, GIRSHICK R, et al. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 39(6): 1137-1149. doi: 10.1109/TPAMI.2016.2577031
[2]	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint arXiv:1409.1556, 2014.
[3]	LI G, XIE Y, LIN L, et al. Instance-level salient object segmentation[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 2386-2395.
[4]	LI M, XIE Q, ZHAO Q, et al. Video rain streak removal by multiscale convolutional sparse coding[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 6644-6653.
[5]	CHEN J, TAN C H, HOU J, et al. Robust video content alignment and compensation for rain removal in a cnn framework[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2018: 6286-6295.
[6]	GARG K, NAYAR S K. Detection and removal of rain from videos[C]// Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2004, IEEE, 2004, 1:I-I.
[7]	BOSSU J, HAUTIERE N, TAREL J P. Rain or snow detection in image sequences through use of a histogram of orientation of streaks[J]. International journal of computer vision, 2011, 93(3): 348-367. doi: 10.1007/s11263-011-0421-7
[8]	KANG L W, LIN C W, FU Y H. Automatic single-image-based rain streaks removal via image decomposition[J]. IEEE transactions on image processing, 2011, 21(4): 1742-1755. doi: 10.1109/TIP.2011.2179057
[9]	LUO Y, XU Y, JI H. Removing rain from a single image via discriminative sparse coding[C]// Proceedings of the IEEE international conference on computer vision, 2015: 3397-3405.
[10]	ZHU L, FU C W, LISCHINSKI D, et al. Joint bi-layer optimization for single-image rain streak removal[C]// Proceedings of the IEEE international conference on computer vision, 2017: 2526-2534.
[11]	REN D, ZUO W, HU Q, et al. Progressive image deraining networks: A better and simpler baseline[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 3937-3946.
[12]	WANG T, YANG X, XU K, et al. Spatial attentive single-image deraining with a high quality real rain dataset[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 12270-12279.
[13]	YASARLA R, PATEL V M. Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 8405-8414.
[14]	REYNOLDS D A, QUATIERI T F, DUNN R B. Speaker verification using adapted Gaussian mixture models[J]. Digital signal processing, 2000, 10(1-3): 19-41. doi: 10.1006/dspr.1999.0361
[15]	CHEN Y L, HSU C T. A generalized low-rank appearance model for spatio-temporally correlated rain streaks[C]// Proceedings of the IEEE international conference on computer vision, 2013: 1968-1975.
[16]	YANG W, TAN R T, FENG J, et al. Deep joint rain detection and removal from a single image[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2017: 1357-1366.
[17]	ZHANG H, SINDAGI V, PATEL V M. Image de-raining using a conditional generative adversarial network[J]. IEEE transactions on circuits and systems for video technology, 2019, 30(11): 3943-3956. doi: 10.1109/TCSVT.76
[18]	ZHU H, PENG X, ZHOU J T, et al. Singe image rain removal with unpaired information: A differentiable programming perspective[C]// Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33(1): 9332-9339.
[19]	CHEN D, HE M, FAN Q, et al. Gated context aggregation network for image dehazing and deraining[C]// 2019 IEEE winter conference on applications of computer vision (WACV), IEEE, 2019: 1375-1383.
[20]	PU J, CHEN X, ZHANG L, et al. Removing rain based on a cycle generative adversarial network[C]// 2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA), IEEE, 2018: 621-626.
[21]	WANG C, XING X, WU Y, et al. Dcsfn: Deep cross-scale fusion network for single image rain removal[C]// Proceedings of the 28th ACM international conference on multimedia, 2020: 1643-1651.
[22]	JIANG K, WANG Z, YI P, et al. Multi-scale progressive fusion network for single image deraining[C]// Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020: 8346-8355.
[23]	LIN C Y, TAO Z, XU A S, et al. Sequential dual attention network for rain streak removal in a single image[J]. IEEE Transactions on Image Processing, 2020, 29: 9250-9265. doi: 10.1109/TIP.83
[24]	LI Y, TAN R T, GUO X, et al. Rain streak removal using layer priors[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2016: 2736-2744.
[25]	FU X, HUANG J, DING X, et al. Clearing the skies: A deep network architecture for single-image rain removal[J]. IEEE Transactions on Image Processing, 2017, 26(6): 2944-2956. doi: 10.1109/TIP.2017.2691802 pmid: 28410108
[26]	HE K, ZHANG X, REN S, et al. Identity mappings in deep residual networks[C]// European conference on computer vision, Springer, Cham, 2016: 630-645.
[27]	ZHU H, WANG C, ZHANG Y, et al. Physical model guided deep image deraining[C]// 2020 IEEE International Conference on Multimedia and Expo (ICME), IEEE, 2020: 1-6.
[28]	YANG Y, LU H. Single image deraining using a recurrent multi-scale aggregation and enhancement network[C]// 2019 IEEE International Conference on Multimedia and Expo (ICME), IEEE, 2019: 1378-1383.
[29]	ZHANG Y, ZHANG J, HUANG B, et al. Single-image deraining via a recurrent memory unit network[J]. Knowledge-Based Systems, 2021, 218: 106832. doi: 10.1016/j.knosys.2021.106832
[30]	REN D, ZUO W, HU Q, et al. Progressive image deraining networks: A better and simpler baseline[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 3937-3946.
[31]	GUO M, HUANG B, ZHANG J, et al. DFBDehazeNet: an end-to-end dense feedback network for single image dehazing[J]. Journal of Electronic Imaging, 2021, 30(3): 033004.
[32]	GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]// International Conference on Neural Information Processing Systems, 2014: 2672-2680.
[33]	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// Proceedings of the IEEE international conference on computer vision, 2017: 2223-2232.
[34]	LU B, CHEN J C, CHELLAPPA R. Unsupervised domain-specific deblurring via disentangled representations[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 10225-10234.
[35]	ENGIN D, GENÇ A, KEMAL EKENEL H. Cycle-dehaze: Enhanced cyclegan for single image dehazing[C]// Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2018: 825-833.
[36]	YUAN Y, LIU S, ZHANG J, et al. Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018: 701-710.
[37]	PU J, CHEN X, ZHANG L, et al. Removing rain based on a cycle generative adversarial network[C]// 2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA), IEEE, 2018: 621-626.
[38]	LI X, WU J, LIN Z, et al. Recurrent squeeze-and-excitation context aggregation net for single image deraining[C]// Proceedings of the European conference on computer vision (ECCV), 2018: 254-269.
[39]	QIAN R, TAN R T, YANG W, et al. Attentive generative adversarial network for raindrop removal from a single image[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2018: 2482-2491.
[40]	WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE transactions on image processing, 2004, 13(4): 600-612. doi: 10.1109/tip.2003.819861 pmid: 15376593
[41]	IGNATOV A, KOBYSHEV N, TIMOFTE R, et al. Dslr-quality photos on mobile devices with deep convolutional networks[C]// Proceedings of the IEEE International Conference on Computer Vision, 2017: 3277-3285.
[42]	ZHANG H, PATEL V M. Density-aware single image de-raining using a multi-stream dense network[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2018: 695-704.
[43]	FU X, HUANG J, ZENG D, et al. Removing rain from single images via a deep detail network[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2017: 3855-3863.
[44]	WEI W, MENG D, ZHAO Q, et al. Semi-supervised transfer learning for image rain removal[C]// Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019: 3877-3886.
[45]	CHEN L, LU X, ZHANG J, et al. HINet: Half instance normalization network for image restoration[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 182-192.

Methods	Rain100H			Rain100L			Rain800			Rain12000
Methods	SSIM	PSNR		SSIM	PSNR		SSIM	PSNR		SSIM	PSNR
DerainNet^[25] DetailNet^[43] JORDER^[16] RESCAN^[38] PReNet^[30] SIRR^[44] MSPFN^[22] CycleGAN^[39] RR-GAN^[18] HINet^[45] CDNet(ours)	0.78 — 0.74 0.84 0.89 — 0.86 — — 0.89 0.91	24.95 — 23.45 25.92 27.89 — 28.66 — — 30.65 30.77		0.94 0.92 0.97 0.96 0.98 0.93 0.93 0.83 — 0.97 0.98	32.04 32.38 36.11 36.12 36.69 32.37 32.40 24.61 — 37.28 37.93		0.73 0.73 0.78 0.84 0.85 — 0.84 0.82 0.76 — 0.85	21.16 21.16 22.24 24.09 26.76 — 26.43 23.95 23.51 — 26.85		0.90 — 0.87 0.89 0.92 — 0.92 — — 0.91 0.92	27.33 — 29.75 32.35 32.28 — 32.29 — — 33.05 33.19

基于多尺度和循环生成对抗的连接式去雨网络

Connected Deraining Network Based on Multi-Scale and Cyclic Generative Adversarial

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 45

相关文章 2

编辑推荐

Metrics

本文评价

[1]	陈栋, 李明, 陈淑文. 结合Transformer和多层特征聚合的高光谱图像分类算法[J]. 数据与计算发展前沿, 2023, 5(3): 138-151.
[2]	石晓霖,毕重科,黄元琪,邓亮,王岳青,王昉. 基于虚拟视点的大规模数值模拟的可交互式原位可视化[J]. 数据与计算发展前沿, 2021, 3(4): 30-43.