A Survey on Few-Shot Image Semantic Segmentation

doi:10.11871/jfdc.issn.2096-742X.2021.06.002

Abstract

Abstract:

[Objective] This paper introduces the few-shot image semantic segmentation methods systematically and comprehensively, as a reference to the design of the few-shot segmentation algorithm.[Methods] The metric-based meta-learning method is employed to perform the few-shot segmentation tasks. According to whether the metric tool is learnable, the few-shot segmentation algorithms are divided into prototype-based methods and parameter-based methods. This paper describes the advantages and disadvantages of both algorithms. [R-esults]Some classical and recent research works about the few-shot image semantic segmentation are analyzed in detail, together with an introduction of the main applications of the few-shot segmentation algorithm. [Conclusions] Hereinafter, the future development direction and trend of few-shot image segmentation are discussed, and its key problems and challenges are analyzed.

Key words: computer vision, image data, deep learning, image segmentation, few-shot learning, few-shot segmentation

CHEN Qiong,YANG Yong,HUANG Tianlin,FENG Yuan. A Survey on Few-Shot Image Semantic Segmentation[J]. Frontiers of Data and Computing, 2021, 3(6): 17-34.

Figures/Tables 9

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Table 1

Table 2

Table 3

References 74

[1]	Dong G, Yan Y, Shen C, et al. Real-time high-perfo-rmance semantic image segmentation of urban street scenes[J]. IEEE Transactions on Intelligent Transpor-tation Systems, 2020,22(6):3258-3274.
[2]	Badrinarayanan V, Kendall A, Cipolla R. Segnet: A deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE transactions on pattern analysis and machine intelligence, 2017,39(12):2481-2495.
[3]	Shaban A, Bansal S, Liu Z, et al. One-Shot Learning for Semantic Segmentation[C]//British Machine Vision Conference, 2017: 167. 1- 167.13.
[4]	Vinyals O, Blundell C, Lillicrap T, et al. Matching net-works for one shot learning[J]. Advances in neural infor-mation processing systems, 2016,29:3630-3638.
[5]	Huisman M, van Rijn J N, Plaat A. A survey of deep meta-learning[J]. Artificial Intelligence Review, 2021: 1-59.
[6]	Boudiaf M, Kervadec H, Masud Z I, et al. Few-Shot Seg-mentation Without Meta-Learning: A Good Transductive Inference Is All You Need? [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 13979-13988.
[7]	Hospedales T, Antoniou A, Micaelli P, et al. Meta-learning in neural networks: A survey[J]. arXiv preprint arXiv: 2004. 05439, 2020.
[8]	Vanschoren J. Meta-learning: A survey[J]. arXiv preprint arXiv: 1810. 03548, 2018.
[9]	Chen W Y, Liu Y C, Kira Z, et al. A closer look at few-shot classification[J]. arXiv preprint arXiv: 1904. 04232, 2019.
[10]	Xu W, Wang H, Tu Z. Attentional Constellation Nets for Few-Shot Learning[C/OL]. International Conference on Learning Representations, 2021 -06-23[2021-12-11].https://openreview.net/pdf?id=vujTf_I8Kmc.
[11]	Min J, Kang D, Cho M. Hypercorrelation squeeze for few-shot segmentation[J]. arXiv preprint arXiv: 2104. 01538, 2021.
[12]	Wu Z, Shi X, Lin G, et al. Learning meta-class memory for few-shot semantic segmentation [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 517-526.
[13]	Zhu C, Chen F, Ahmed U, et al. Semantic relation reaso-ning for shot-stable few-shot object detection [C]//Proce-edings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 8782-8791.
[14]	Hu H, Bai S, Li A, et al. Dense Relation Distillation with Context-aware Aggregation for Few-Shot Object Detec-tion [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 10185-10194.
[15]	Sun B, Li B, Cai S, et al. FSCE: Few-shot object detec-tion via contrastive proposal encoding [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 7352-7362.
[16]	Lai X, Tian Z, Jiang L, et al. Semi-Supervised Semantic Segmentation With Directional Context-Aware Consistency [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 1205-1214.
[17]	Zhong Y, Yuan B, Wu H, et al. Pixel Contrastive-Con-sistent Semi-Supervised Semantic Segmentation [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 7273-7282.
[18]	Chang Y T, Wang Q, Hung W C, et al. Weakly-supervised semantic segmentation via sub-category exploration [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 8991-9000.
[19]	Zhang M, Zhou Y, Zhao J, et al. A survey of semi-and weakly supervised semantic segmentation of images[J]. Artificial Intelligence Review, 2020,53(6):4259-4288.
[20]	Wang C, Farazi M, Barnes N. Recursive Training for Zero-Shot Semantic Segmentation [C]//2021 International Joint Conference on Neural Networks (IJCNN), IEEE, 2021: 1-8.
[21]	Bucher M, Vu T H, Cord M, et al. Zero-shot semantic segmentation[J]. Advances in Neural Information Proces-sing Systems, 2019,32:468-479.
[22]	Rakelly K, Shelhamer E, Darrell T, et al. Conditional networks for few-shot semantic segmentation[J/OL]. 2018 -04-4[2021-12-11]. https://openreview.net/pdf?id=SkMjFKJwG.
[23]	Dong N, Xing E P. Few-shot semantic segmentation with prototype learning [C]//BMVC, 2018,3(4):79.
[24]	Tian P, Wu Z, Qi L, et al. Differentiable meta-learning model for few-shot semantic segmentation [C]//Pro-ceedings of the AAAI Conference on Artificial Intel-ligence. 2020,34(07):12087-12094.
[25]	Yang Y, Meng F, Li H, et al. A new local transformation module for few-shot segmentation [C]//International Con-ference on Multimedia Modeling. Springer, Cham, 2020: 76-87.
[26]	Yang B, Liu C, Li B, et al. Prototype mixture models for few-shot semantic segmentation [C]//European Confe-rence on Computer Vision. Springer, Cham, 2020: 763-778.
[27]	Bhunia A K, Bhunia A K, Ghose S, et al. A deep one-shot network for query-based logo retrieval[J]. Pattern Recognition, 2019,96:106965.
[28]	Nguyen K, Todorovic S. Feature weighting and boosting for few-shot segmentation [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 622-631.
[29]	Snell J, Swersky K, Zemel R S. Prototypical networks for few-shot learning[J]. arXiv preprint arXiv: 1703. 05175, 2017.
[30]	Li G, Jampani V, Sevilla-Lara L, et al. Adaptive Prototype Learning and Allocation for Few-Shot Segmentation [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 8334-8343.
[31]	Liu Y, Zhang X, Zhang S, et al. Part-aware prototype network for few-shot semantic segmentation [C]//Eur-opean Conference on Computer Vision. Springer, Cham, 2020: 142-158.
[32]	Zhang C, Lin G, Liu F, et al. Canet: Class-agnostic se-gmentation networks with iterative refinement and atten-tive few-shot learning [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 5217-5226.
[33]	Tian Z, Zhao H, Shu M, et al. Prior guided feature enri-chment network for few-shot segmentation[J]. IEEE Tran-sactions on Pattern Analysis & Machine Intelligence, 2020 ( 01):1-1.
[34]	Yang X, Wang B, Chen K, et al. Brinet: Towards brid-ging the intra-class and inter-class gaps in one-shot seg-mentation[J]. arXiv preprint arXiv: 2008. 06226, 2020.
[35]	He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C]//Proceedings of the IEEE conference on computer vision and pattern recognition, 2016: 770-778.
[36]	Liu W, Zhang C, Lin G, et al. Crnet: Cross-reference networks for few-shot segmentation [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 4165-4173.
[37]	Chen L C, Papandreou G, Kokkinos I, et al. Semantic image segmentation with deep convolutional nets and fully connected crfs[J]. arXiv preprint arXiv: 1412. 7062, 2014.
[38]	Chen L C, Papandreou G, Kokkinos I, et al. Semantic image segmentation with deep convolutional nets and fully connected crfs[J]. arXiv preprint arXiv: 1412. 7062, 2014.
[39]	Chen L C, Zhu Y, Papandreou G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation [C]//Proceedings of the European confe-rence on computer vision (ECCV), 2018: 801-818.
[40]	Zhao H, Shi J, Qi X, et al. Pyramid scene parsing network [C]//Proceedings of the IEEE conference on computer vision and pattern recognition, 2017: 2881-2890.
[41]	Zhang B, Xiao J, Qin T. Self-Guided and Cross-Guided Learning for Few-Shot Segmentation [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 8312-8321.
[42]	Xie G S, Liu J, Xiong H, et al. Scale-Aware Graph Neural Network for Few-Shot Semantic Segmentation [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 5475-5484.
[43]	Tao A, Sapra K, Catanzaro B. Hierarchical multi-scale attention for semantic segmentation[J]. arXiv preprint arXiv: 2005. 10821, 2020.
[44]	Huang Z, Wang X, Huang L, et al. Ccnet: Criss-cross attention for semantic segmentation [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019: 603-612.
[45]	Zhang H, Zhang H, Wang C, et al. Co-occurrent features in semantic segmentation [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog-nition, 2019: 548-557.
[46]	Yuan Y, Chen X, Wang J. Object-contextual repre-sentations for semantic segmentation[C]//Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16. Springer International Publishing, 2020: 173-190.
[47]	Li X, Zhong Z, Wu J, et al. Expectation-maximization attention networks for semantic segmentation [C]//Procee-dings of the IEEE/CVF International Conference on Com-puter Vision, 2019: 9167-9176.
[48]	Zhang C, Lin G, Liu F, et al. Pyramid graph networks with connection attentions for region-based one-shot semantic segmentation [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019: 9587-9595.
[49]	Wang X, Girshick R, Gupta A, et al. Non-local neural networks [C]//Proceedings of the IEEE conference on computer vision and pattern recognition, 2018: 7794-7803.
[50]	Xie G S, Xiong H, Liu J, et al. Few-shot semantic segm-entation with cyclic memory network [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 7293-7302.
[51]	Lu Z, He S, Zhu X, et al. Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transfor-mer [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 8741-8750.
[52]	Siam M, Oreshkin B N, Jagersand M. Amp: Adaptive masked proxies for few-shot segmentation [C]//Procee-dings of the IEEE/CVF International Conference on Computer Vision, 2019: 5249-5258.
[53]	Zhang X, Wei Y, Yang Y, et al. Sg-one: Similarity gui-dance network for one-shot semantic segmentation[J]. IEEE transactions on cybernetics, 2020,50(9):3855-3865.
[54]	Wang K, Liew J H, Zou Y, et al. Panet: Few-shot image semantic segmentation with prototype alignment [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019: 9197-9206.
[55]	Zhang X, Wei Y, Li Z, et al. Rich Embedding Features for One-Shot Semantic Segmentation[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021: 1-10.
[56]	Gairola S, Hemani M, Chopra A, et al. Simpropnet: Improved similarity propagation for few-shot image segmentation[J]. arXiv preprint arXiv: 2004. 15014, 2020.
[57]	Kim A. Fast slic[EB/OL].https://github.com/Algy/fast-slic.
[58]	Yang L, Zhuo W, Qi L, et al. Mining Latent Classes for Few-shot Segmentation[J]. arXiv preprint arXiv: 2103. 15402, 2021.
[59]	Ouyang C, Biffi C, Chen C, et al. Self-supervision with superpixels: Training few-shot medical image segmen-tation without annotation [C]//European Conference on Computer Vision. Springer, Cham, 2020: 762-780.
[60]	Feyjie A R, Azad R, Pedersoli M, et al. Semi-supervised few-shot learning for medical image segmentation[J]. arXiv preprint arXiv: 2003. 08462, 2020.
[61]	Guo Y, Wang H, Hu Q, et al. Deep Learning for 3D Point Clouds: A Survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020,99:1-1.
[62]	王文曦, 李乐林. 深度学习在点云分类中的研究综述[J/OL]. 计算机工程与应用:1-17[ 2021- 10- 25].http://kns.cnki.net/kcms/detail/11.2127.TP.20211021.1009.004.html.
[63]	Landrieu L, Simonovsky M. Large-scale point cloud semantic segmentation with superpoint graphs [C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 4558-4567.
[64]	Hu Q, Yang B, Xie L, et al. Randla-net: Efficient semantic segmentation of large-scale point clouds [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 11108-11117.
[65]	Qi C R, Su H, Mo K, et al. Pointnet: Deep learning on point sets for 3d classification and segmentation [C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 652-660.
[66]	李梦怡, 朱定局. 基于全卷积网络的图像语义分割方法综述[J]. 计算机系统应用, 2021,30(09):41-52.
[67]	刘念. 基于小样本深度学习的遥感地物分类与分割方法研究[D]. 重庆大学, 2019.
[68]	Pan X, Shi J, Luo P, et al. Spatial as deep: Spatial cnn for traffic scene understanding [C]//Thirty-Second AAAI Conference on Artificial Intelligence. 2018: 7276— 7283.
[69]	Wang Z, Ren W, Qiu Q. Lanenet: Real-time lane detec-tion networks for autonomous driving[J]. arXiv preprint arXiv: 1807. 01726, 2018.
[70]	Hou Y, Ma Z, Liu C, et al. Learning lightweight lane detection cnns by self attention distillation [C]//Procee-dings of the IEEE/CVF international conference on computer vision, 2019: 1013-1021.
[71]	Everingham M, Van Gool L, Williams C K I, et al. The pascal visual object classes (voc) challenge[J]. Interna-tional journal of computer vision, 2010,88(2):303-338.
[72]	Hariharan B, Arbeláez P, Bourdev L, et al. Semantic contours from inverse detectors [C]//2011 International Conference on Computer Vision. IEEE, 2011: 991-998.
[73]	Lin T Y, Maire M, Belongie S, et al. Microsoft coco: Common objects in context [C]//European conference on computer vision. Springer, Cham, 2014: 740-755.
[74]	Li X, Wei T, Chen Y P, et al. Fss-1000: A 1000-class dataset for few-shot segmentation [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 2869-2878.

模型	方法	骨干网络	Mean-IoU (1-shot)	FB-IoU (1-shot)	Mean-IoU (5-shot)	FB-IoU (5-shot)
PANet^[54] SG-One^[53] FWB^[28] PPNet^[31] SimPropNet^[56] SAGNN^[42] MLC^[58] ASGNet^[30]	原型原型原型原型原型原型原型原型	VGG-16 VGG-16 ResNet-101 ResNet-50 ResNet-50 ResNet-50 ResNet-50 ResNet-50	48.1 46.3 56.2 51.5 57.2 62.1 62.1 59.3	68.5 63.1 - - 73.0 73.2 69.2	55.7 47.1 59.9 62.0 60.0 62.8 66.1 63.9	70.7 65.9 - - 72.9 73.3 74.2
CANet^[32] PFENet^[33] BriNet^[34] PMMS^[26] PGNet^[48] CRNet^[36] SCL_PFENet^[41] SCL_CANet^[41] CMN^[50] CWT^[51]	参数参数参数参数参数参数参数参数参数参数	ResNet-50 ResNet-50 ResNet-50 ResNet-50 ResNet-50 ResNet-50 ResNet-50 ResNet-50 ResNet-50 ResNet-50	55.4 60.8 57.1 55.2 56.0 55.7 61.8 57.5 62.8 56.4	66.2 73.3 69.9 66.8 71.9 70.3 72.3	57.1 61.9 56.8 58.5 58.8 62.9 59.2 63.7 63.7	69.6 73.9 70.5 71.5 72.8 70.7 72.8

模型	方法	骨干网络	Mean-IoU (1-shot)	FB-IoU (1-shot)	Mean-IoU (5-shot)	FB-IoU (5-shot)
PANet^[54] FWB^[28] PPNet^[31] SAGNN^[42] MLC^[58] ASGNet^[30]	原型原型原型原型原型原型	VGG-16 ResNet-101 ResNet-50 ResNet-101 ResNet-50 ResNet-50	20.9 21.2 29.0 37.2 33.9 34.6	59.2 60.9 60.4	29.7 23.7 38.5 42.9 40.6 42.5	63.5 63.4 67.0
CANet^[32] PFENet^[33] BriNet^[34] PMMS^[26] SCL_PFENet^[41] CMN^[50] CWT^[51]	参数参数参数参数参数参数参数	ResNet-50 ResNet-101 ResNet-50 ResNet-50 ResNet-50 ResNet-50 ResNet-50	32.4 34.4 29.6 37.0 39.3 32.9	58.6 61.7	37.4 34.3 39.9 43.1 41.3	61.9 63.3

模型	方法	骨干网络	参数量
PANet^[54] SG-One^[53] FWB^[28] PPNet^[31]	原型原型原型原型	VGG-16 VGG-16 ResNet-101 ResNet-50	14.7M 19.0M 43.0M 31.5M
CANet^[32] PFENet^[33] BriNet^[34] PMMS^[26] PGNet^[48]	参数参数参数参数参数	ResNet-50 ResNet-50 ResNet-50 ResNet-50 ResNet-50	19.0M 10.8M 20.5M 19.6M 17.2M