Connected Deraining Network Based on Multi-Scale and Cyclic Generative Adversarial

doi:10.11871/jfdc.issn.2096-742X.2023.05.011

Abstract

Abstract:

[Objective] Image deraining can be used as a preprocessing step for computer vision tasks so that the results of computer vision tasks such as automatic drive and target recognition can be improved. [Methods] In this paper, multi-scale information exchange is connected with a cyclic generative adversarial network. The proposed method is divided into two parts according to the training steps. First, the rain streak information is obtained through multi-scale information exchange for initial rain removal. Then the image with initial rain removal is further enhanced by a cyclic generative adversarial network so as to obtain the best rain removal effect. [Results] This method can effectively remove the rain information in the image and restore a clear image. This method presented in this paper has achieved good rain removal results in PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity) evaluation indexes and can better preserve the details of the image. [Conclusions] By comparing the results on synthetic datasets and real images with other image rain removal methods, this method has achieved better results and can provide better support for other computer vision tasks.

Key words: image processing, image deraining, multi-scale information exchange, cycle generative adversarial network

LANG Xiaoqi, ZHANG Juan. Connected Deraining Network Based on Multi-Scale and Cyclic Generative Adversarial[J]. Frontiers of Data and Computing, 2023, 5(5): 128-139, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2023.05.011.

Figures/Tables 7

Fig.1

Table 1

Table 2

Table 3

Table 4

Fig.2

Fig.3

References 45

[1]	REN S, HE K, GIRSHICK R, et al. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 39(6): 1137-1149. doi: 10.1109/TPAMI.2016.2577031
[2]	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint arXiv:1409.1556, 2014.
[3]	LI G, XIE Y, LIN L, et al. Instance-level salient object segmentation[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 2386-2395.
[4]	LI M, XIE Q, ZHAO Q, et al. Video rain streak removal by multiscale convolutional sparse coding[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 6644-6653.
[5]	CHEN J, TAN C H, HOU J, et al. Robust video content alignment and compensation for rain removal in a cnn framework[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2018: 6286-6295.
[6]	GARG K, NAYAR S K. Detection and removal of rain from videos[C]// Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2004, IEEE, 2004, 1:I-I.
[7]	BOSSU J, HAUTIERE N, TAREL J P. Rain or snow detection in image sequences through use of a histogram of orientation of streaks[J]. International journal of computer vision, 2011, 93(3): 348-367. doi: 10.1007/s11263-011-0421-7
[8]	KANG L W, LIN C W, FU Y H. Automatic single-image-based rain streaks removal via image decomposition[J]. IEEE transactions on image processing, 2011, 21(4): 1742-1755. doi: 10.1109/TIP.2011.2179057
[9]	LUO Y, XU Y, JI H. Removing rain from a single image via discriminative sparse coding[C]// Proceedings of the IEEE international conference on computer vision, 2015: 3397-3405.
[10]	ZHU L, FU C W, LISCHINSKI D, et al. Joint bi-layer optimization for single-image rain streak removal[C]// Proceedings of the IEEE international conference on computer vision, 2017: 2526-2534.
[11]	REN D, ZUO W, HU Q, et al. Progressive image deraining networks: A better and simpler baseline[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 3937-3946.
[12]	WANG T, YANG X, XU K, et al. Spatial attentive single-image deraining with a high quality real rain dataset[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 12270-12279.
[13]	YASARLA R, PATEL V M. Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 8405-8414.
[14]	REYNOLDS D A, QUATIERI T F, DUNN R B. Speaker verification using adapted Gaussian mixture models[J]. Digital signal processing, 2000, 10(1-3): 19-41. doi: 10.1006/dspr.1999.0361
[15]	CHEN Y L, HSU C T. A generalized low-rank appearance model for spatio-temporally correlated rain streaks[C]// Proceedings of the IEEE international conference on computer vision, 2013: 1968-1975.
[16]	YANG W, TAN R T, FENG J, et al. Deep joint rain detection and removal from a single image[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2017: 1357-1366.
[17]	ZHANG H, SINDAGI V, PATEL V M. Image de-raining using a conditional generative adversarial network[J]. IEEE transactions on circuits and systems for video technology, 2019, 30(11): 3943-3956. doi: 10.1109/TCSVT.76
[18]	ZHU H, PENG X, ZHOU J T, et al. Singe image rain removal with unpaired information: A differentiable programming perspective[C]// Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33(1): 9332-9339.
[19]	CHEN D, HE M, FAN Q, et al. Gated context aggregation network for image dehazing and deraining[C]// 2019 IEEE winter conference on applications of computer vision (WACV), IEEE, 2019: 1375-1383.
[20]	PU J, CHEN X, ZHANG L, et al. Removing rain based on a cycle generative adversarial network[C]// 2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA), IEEE, 2018: 621-626.
[21]	WANG C, XING X, WU Y, et al. Dcsfn: Deep cross-scale fusion network for single image rain removal[C]// Proceedings of the 28th ACM international conference on multimedia, 2020: 1643-1651.
[22]	JIANG K, WANG Z, YI P, et al. Multi-scale progressive fusion network for single image deraining[C]// Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020: 8346-8355.
[23]	LIN C Y, TAO Z, XU A S, et al. Sequential dual attention network for rain streak removal in a single image[J]. IEEE Transactions on Image Processing, 2020, 29: 9250-9265. doi: 10.1109/TIP.83
[24]	LI Y, TAN R T, GUO X, et al. Rain streak removal using layer priors[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2016: 2736-2744.
[25]	FU X, HUANG J, DING X, et al. Clearing the skies: A deep network architecture for single-image rain removal[J]. IEEE Transactions on Image Processing, 2017, 26(6): 2944-2956. doi: 10.1109/TIP.2017.2691802 pmid: 28410108
[26]	HE K, ZHANG X, REN S, et al. Identity mappings in deep residual networks[C]// European conference on computer vision, Springer, Cham, 2016: 630-645.
[27]	ZHU H, WANG C, ZHANG Y, et al. Physical model guided deep image deraining[C]// 2020 IEEE International Conference on Multimedia and Expo (ICME), IEEE, 2020: 1-6.
[28]	YANG Y, LU H. Single image deraining using a recurrent multi-scale aggregation and enhancement network[C]// 2019 IEEE International Conference on Multimedia and Expo (ICME), IEEE, 2019: 1378-1383.
[29]	ZHANG Y, ZHANG J, HUANG B, et al. Single-image deraining via a recurrent memory unit network[J]. Knowledge-Based Systems, 2021, 218: 106832. doi: 10.1016/j.knosys.2021.106832
[30]	REN D, ZUO W, HU Q, et al. Progressive image deraining networks: A better and simpler baseline[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 3937-3946.
[31]	GUO M, HUANG B, ZHANG J, et al. DFBDehazeNet: an end-to-end dense feedback network for single image dehazing[J]. Journal of Electronic Imaging, 2021, 30(3): 033004.
[32]	GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]// International Conference on Neural Information Processing Systems, 2014: 2672-2680.
[33]	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// Proceedings of the IEEE international conference on computer vision, 2017: 2223-2232.
[34]	LU B, CHEN J C, CHELLAPPA R. Unsupervised domain-specific deblurring via disentangled representations[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 10225-10234.
[35]	ENGIN D, GENÇ A, KEMAL EKENEL H. Cycle-dehaze: Enhanced cyclegan for single image dehazing[C]// Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2018: 825-833.
[36]	YUAN Y, LIU S, ZHANG J, et al. Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018: 701-710.
[37]	PU J, CHEN X, ZHANG L, et al. Removing rain based on a cycle generative adversarial network[C]// 2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA), IEEE, 2018: 621-626.
[38]	LI X, WU J, LIN Z, et al. Recurrent squeeze-and-excitation context aggregation net for single image deraining[C]// Proceedings of the European conference on computer vision (ECCV), 2018: 254-269.
[39]	QIAN R, TAN R T, YANG W, et al. Attentive generative adversarial network for raindrop removal from a single image[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2018: 2482-2491.
[40]	WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE transactions on image processing, 2004, 13(4): 600-612. doi: 10.1109/tip.2003.819861 pmid: 15376593
[41]	IGNATOV A, KOBYSHEV N, TIMOFTE R, et al. Dslr-quality photos on mobile devices with deep convolutional networks[C]// Proceedings of the IEEE International Conference on Computer Vision, 2017: 3277-3285.
[42]	ZHANG H, PATEL V M. Density-aware single image de-raining using a multi-stream dense network[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2018: 695-704.
[43]	FU X, HUANG J, ZENG D, et al. Removing rain from single images via a deep detail network[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2017: 3855-3863.
[44]	WEI W, MENG D, ZHAO Q, et al. Semi-supervised transfer learning for image rain removal[C]// Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019: 3877-3886.
[45]	CHEN L, LU X, ZHANG J, et al. HINet: Half instance normalization network for image restoration[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 182-192.

Methods	Rain100H			Rain100L			Rain800			Rain12000
Methods	SSIM	PSNR		SSIM	PSNR		SSIM	PSNR		SSIM	PSNR
DerainNet^[25] DetailNet^[43] JORDER^[16] RESCAN^[38] PReNet^[30] SIRR^[44] MSPFN^[22] CycleGAN^[39] RR-GAN^[18] HINet^[45] CDNet(ours)	0.78 — 0.74 0.84 0.89 — 0.86 — — 0.89 0.91	24.95 — 23.45 25.92 27.89 — 28.66 — — 30.65 30.77		0.94 0.92 0.97 0.96 0.98 0.93 0.93 0.83 — 0.97 0.98	32.04 32.38 36.11 36.12 36.69 32.37 32.40 24.61 — 37.28 37.93		0.73 0.73 0.78 0.84 0.85 — 0.84 0.82 0.76 — 0.85	21.16 21.16 22.24 24.09 26.76 — 26.43 23.95 23.51 — 26.85		0.90 — 0.87 0.89 0.92 — 0.92 — — 0.91 0.92	27.33 — 29.75 32.35 32.28 — 32.29 — — 33.05 33.19