基于隐式分区学习深度特征融合重建曲面网络

doi:10.11871/jfdc.issn.2096-742X.2024.06.003

摘要/Abstract

摘要：

【目的】在计算机视觉和图形学领域中，原始点云曲面重建任务仍具挑战性。目前大多数研究在隐式空间中学习各种特征并直接融合，但这种方法很难准确解释3D模型，并且重建模型存在特征消失导致的不完整曲面。【方法】为解决这个问题，本文引入一种新的隐式表示方法，将特征划分为全局特征和局部特征，首先学习分块后局部特征，局部点云在潜在隐式空间中学习局部特征快速准确获取点云特征，然后将各部分深度特征隐式融合，将学习后的全局特征进行融合并重建曲面模型。【结果】该方法将一个3D全局形状转化为多个局部形状进行建模，局部形状通过深度特征提取划分全局统一，能够更有效地提取3D形状的隐式曲面，从而重建3D曲面。将其命名为隐式分区深度特征融合（Implicit Partition Deep Feature Fusion, IPDFF）。【局限】IPDFF模型虽然对复杂模型适用效果较好，但对复杂区域缺失点云或均为复杂特征点云重建效果欠佳。【结论】在实验结果中，IPDFF在视觉重建效果和几个定量指标上优于其他基线方法，重建曲面后有较强的鲁棒性和实用性，且模型具有更强的细节特征。

关键词: 三维图像处理, 曲面重建, 深度学习, 点云特征融合, 点云特征提取

Abstract:

[Objective] In the field of computer vision and graphics, the task of original point cloud surface reconstruction is still challenging. Most current studies learn various features in implicit space and fuse them directly. But this method is difficult to accurately interpret 3D models, and there are incomplete surfaces caused by the disappearance of features in the reconstructed models. [Methods] To solve this problem, this paper introduces a new implicit representation method, which divides features into global features and local features. First, the local features after segmentation are learned, and the local point cloud learns local features in the potential implicit space to quickly and accurately obtain the point cloud features. Then, the deep features of each part are implicitly fused, the learned global features are fused and the surface model is reconstructed. [Results] The method converts a 3D global shape into multiple local shapes for modeling. The local shapes are divided into global unity through deep feature extraction, and the implicit surfaces of 3D shapes can be extracted more effectively, so as to reconstruct 3D surfaces. We will name it Implicit Partition Deep Feature Fusion (IPDFF). [Limitations] Although the IPDFF model is suitable for complex models, it is not effective for the reconstruction of missing point clouds in complex regions or point clouds with complex features. [Conclusions] In the experimental results, IPDFF is superior to other baseline methods in visual reconstruction effect and several quantitative indexes, and the reconstructed surface has strong robustness and practicability, and the model has stronger detail features.

Key words: three-dimensional image processing, surface reconstruction, deep learning, point cloud feature fusion, point cloud feature extraction

卢成浩,陈秀宏. 基于隐式分区学习深度特征融合重建曲面网络[J]. 数据与计算发展前沿, 2024, 6(6): 19-31.

LU Chenghao,CHEN Xiuhong. IPDFF: Reconstructed Surface Network Based on Implicit Partition Learning Deep Feature Fusion[J]. Frontiers of Data and Computing, 2024, 6(6): 19-31, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2024.06.003.

图/表 10

图1

表1

图2

图3

图4

表2

图5

图6

图7

表3

参考文献 55

[1]	ARENS C. The Bowyer-Watson algorithm[J]. An efficient Implementation in a Database Environment, 2002: 5-9.
[2]	AURENHAMMER F, KLEIN R. Voronoi Diagrams[J]. Handbook of Computational Geometry, 2000, 5(10): 201-290.
[3]	RAKOTOSAONA M J, GUERRERO P, AIGERMAN N, et al. Learning delaunay surface elements for mesh reconstruction[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 22-31.
[4]	LUO Y, MI Z, TAO W. Deepdt: Learning geometry from delaunay triangulation for surface reconstruction[C]// Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 35(3): 2277-2285.
[5]	ZHANG Y, LIU Y, ZHANG H, et al. Multimodal remote sensing image matching combining learning features and Delaunay triangulation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-17.
[6]	MENG W, BO P, ZHANG X, et al. An efficient algorithm for approximate Voronoi diagram construction on triangulated surfaces[J]. Computational Visual Media, 2023, 9(3): 443-459.
[7]	WANG L, HAN J, LU S, et al. Rapid reconstruction of point cloud implicit surface[J]. Laser & Optoelectronics Progress, 2021, 58(4): 0415003.
[8]	FUSCHLER M A, BOLLES R C. Random Consensus: A paradigm for model fitting with applications to image analysis and automated cartigraphy[J]. Communications of the ACM, 1981, 24(6): 381-395.
[9]	CUN X, WANG X G. Research and application of moving least square surface fitting subpixel displacement algorithm[J]. Computer Application Research, 2020, 37(S2): 330-332.
[10]	TAMILMATHI A, CHITHRA P L. Tensor block-wise singular value decomposition for 3D point cloud compression[J]. Multimedia Tools and Applications, 2022, 81(26): 37917-37938.
[11]	MA B, LIU Y, HAN Z. Reconstructing surfaces for sparse point clouds with On-Surface Priors[C]. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022: 6305-6315.
[12]	KAZHDAN M, BOLITHO M, HOPPE H. Poisson surface reconstruction[C]// Proceedings of the fourth Eurographics Symposium on Geometry processing, 2006, 7(4).
[13]	KAZHDAN M, HOPPE H. Screened poisson surface reconstruction[J]. Acm Transactions on Graphics, 2013, 32(3): 1-13.
[14]	ESTELLERS V, SCOTT M, TEW K, et al. Robust poisson surface reconstruction[C]// Scale Space and Variational Methods in Computer Vision:5th International Conference, SSVM 2015, Lège-Cap Ferret, France, May 31-June 4, 2015, Proceedings 5. Springer International Publishing, 2015: 525-537.
[15]	HOU F, WANG C, WANG W, et al. Iterative Poisson surface reconstruction (IPSR) for unoriented points[J]. ACM Transactions on Graphics, 2022, 41(4):1-13.
[16]	PREINER R, BOUBEKEUR T, WIMMER M. Gaussian-product subdivision surfaces[J]. ACM Tran-sactions on Graphics (TOG), 2019, 38(4): 1-11.
[17]	LIN S, XIAO D, SHI Z, et al. Surface reconstruction from point clouds without normals by parametrizing the gauss formula[J]. ACM Transactions on Graphics, 2022, 42(2): 1-19.
[18]	PARK J J, FLORENCE P, STRAUB J, et al. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019: 165-174.
[19]	ATZMON M, LIPMAN Y. SALD: Sign Agnostic Learning with Derivatives[C]// International Conference on Learning Representations, 2020.
[20]	MI Z, LUO Y, TAO W. Ssrnet: Scalable 3d surface reconstruction network[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 970-979.
[21]	SITZMANN V, MARTEL J, BERGMAN A, et al. Implicit neural representations with periodic activation functions[J]. Advances in neural information processing systems, 2020, 33: 7462-7473.
[22]	BASHER A, BOUTELLIER J. DC-DFFN: Densely connected deep feature fusion network with sign agnostic learning for implicit shape representation[J]. IEEE Access, 2023, 11: 46399-46412.
[23]	YAVARTANOO M, CHUNG J, NESHATAVAR R, et al. 3dias: 3d shape reconstruction with implicit algebraic surfaces. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021:12446-12455.
[24]	CHEN C, LIU Y, HAN Z. Latent partition implicit with surface codes for 3d representation[C]// European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 322-343.
[25]	QI C, SU H, MO K, et al. Pointnet: Deep learning on point sets for 3d classification and segmentation[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2017: 652-660.
[26]	HU P, HO E, MUNTEANU A. 3DBodyNet: fast reconstruction of 3D animatable human body shape from a single commodity depth camera[J]. IEEE Transactions on Multimedia, 2021, 24: 2139-2149.
[27]	DING C, ZHANG L, CHEN H, et al. Sparsity-based Human Activity Recognition with PointNet using a Portable FMCW Radar[J]. IEEE Internet of Things Journal, 2023, 10(11): 10024-10037.
[28]	MILDENHALL B, SRINIVASAN P, TANCIK M, et al. Nerf: Representing scenes as neural radiance fields for view synthesis[J]. Communications of the ACM, 2021, 65(1): 99-106.
[29]	RAN Y, ZENG J, HE S, et al. NeurAR: Neural Uncertainty for Autonomous 3D Reconstruction With Implicit Neural Representations[J]. IEEE Robotics and Automation Letters, 2023, 8(2): 1125-1132.
[30]	PARK J J, FLORENCE P, STRAUB J, et al. Deepsdf: Learning continuous signed distance functions for shape representation[C]// Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019: 165-174.
[31]	ZHANG R, GUO Z, GAO P, et al. Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training[J]. Advances in neural information processing systems, 2022, 35: 27061-27074.
[32]	CAI X, CAO Y, REN Y, et al. Multi-objective evolutionary 3D face reconstruction based on improved encoder-decoder network[J]. Information Sciences, 2021, 581: 233-248.
[33]	SAYED M, GIBSON J, WATSON J, et al. SimpleRecon: 3D reconstruction without 3D convolutions[C]// European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 1-19.
[34]	MILDENHALL B, SRINIVASAN P P, TANCIK M, et al. Nerf: Representing scenes as neural radiance fields for view synthesis[J]. Communications of the ACM, 2021, 65(1): 99-106.
[35]	WU C Y, JOHNSON J, MALIK J, et al. Multiview compressive coding for 3D reconstruction[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 9065-9075.
[36]	MAZZACCA G, KARAMI A, RIGON S, et al. NeRF for heritage 3D reconstruction[J]. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2023, 48( M-2-2023): 1051-1058.
[37]	QI C R, SU H, MO K, et al. Pointnet: Deep learning on point sets for 3d classification and segmentation[C]// Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 652-660.
[38]	QI C R, YI L, SU H, et al. Pointnet++: Deep hierarchical feature learning on point sets in a metric space[J]. Advances in neural information processing systems, 2017, 30.
[39]	WANG Q, LEI H, QIAN W. Siamese PointNet: 3D Head Pose Estimation with Local Feature Descriptor[J]. Electronics, 2023, 12(5): 1194.
[40]	HUANG G, LIU Z, MAATEN L, et al. Densely connected convolutional networks[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2017: 4700-4708.
[41]	ZHANG H, PATEL V. Densely connected pyramid dehazing network[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2018: 3194-3203.
[42]	PANDEY A, WANG D L. Densely connected neural network with dilated convolutions for real-time speech enhancement in the time domain[C]// ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2020: 6629-6633.
[43]	ZHANG F, FANG J, WAH B, et al. Deep fusionnet for point cloud semantic segmentation[C]// Computer Vision-ECCV 2020: 16th European Conference, Gla-sgow, UK, August 23-28, 2020, Proceedings, Part XXIV 16. Springer International Publishing, 2020: 644-663.
[44]	CHENG R, RAZANI R, TAGHAVI E, et al. 2-s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network[C]// Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 12547-12556.
[45]	XIAO A, YANG X, LU S, et al. FPS-Net: A convolutional fusion network for large-scale LiDAR point cloud segmentation[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 176: 237-249.
[46]	PASCHALIDOU D, GOOL L, GEIGER A. Learning unsupervised hierarchical part decomposition of 3d objects from a single rgb image[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 1060-1070.
[47]	SHARMA G, DASH B, GADELHA M, et al. Surfit: learning to fit surfaces improves few shot learning on point clouds[J]. CoRR 2021, abs/2112.13942.
[48]	LI Y, HAN G, LIU X. Dcnet: Densely connected deep convolutional encoder-decoder network for nasopharyngeal carcinoma segmentation[J]. Sensors, 2021, 21(23): 7877.
[49]	JIANG C, SUD A, MAKADIA A, et al. Local implicit grid representations for 3d scenes[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 6001-6010.
[50]	TAKIKAWA T, LITALIEN J, YIN K, et al. Neural geometric level of detail: Real-time rendering with implicit 3d shapes[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 11358-11367.
[51]	SITZMANN V, MARTEL J, BERGMAN A, et al. Implicit neural representations with periodic activation functions[J]. Advances in neural information processing systems, 2020, 33: 7462-7473.
[52]	BOGO F, ROMERO J, LOPER M, et al. FAUST: Dataset and evaluation for 3D mesh registration[C]// Proceedings of the IEEE conference on computer vision and pattern recognition. 2014: 3794-3801.
[53]	SUN X, WU J, ZHANG X, et al. Pix3d: Dataset and methods for single-image 3d shape modeling[C]// Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 2974-2983.
[54]	KINGA D, ADAM J. A method for stochastic optimization[C]// International conference on learning representations (ICLR), 2015, 5: 6.
[55]	LIN S, XIAO D, SHI Z, et al. Surface Reconstruction from Point Clouds without Normals by Parametrizing the Gauss Formula[J]. ACM Transactions on Graphics, 2022, 42(2): 1-19.

	Gauss		SALD		DC-DFFN		IPDFF
	L1CD	F-Score	L1CD	F-Score	L1CD	F-Score	L1CD	F-Score
Model1	6.514e^-5	0.798	1.980e^-4	0.615	6.275e^-5	0.787	2.416e^-5	0.885
Model2	6.238e^-5	0.791	4.620e^-5	0.828	3.156e^-5	0.826	1.903e^-5	0.909
Model3	8.518e^-5	0.807	8.165e^-5	0.778	6.579e^-5	0.809	4.392e^-5	0.886
Model4	8.166e^-5	0.814	8.238e^-5	0.829	3.166e^-5	0.862	2.829e^-5	0.891
Model5	6.522e^-5	0.782	4.986e^-5	0.840	2.922e^-5	0.852	2.132e^-5	0.900
Model6	6.522e^-5	0.799	5.683e^-5	0.821	3.371e^-5	0.880	2.013e^-5	0.913
Model7	5.396e^-5	0.786	6.708e^-5	0.809	3.697e^-5	0.865	2.503e^-5	0.888
Model8	8.708e^-5	0.817	5.409e^-5	0.825	4.567e^-5	0.863	2.659e^-5	0.897
Model9	6.086e^-5	0.784	4.185e^-5	0.842	3.376e^-5	0.856	2.150e^-5	0.898
Model10	9.619e^-5	0.825	6.077e^-5	0.819	4.192e^-5	0.842	2.980e^-5	0.904

	LPI	DC-DFFN	IPDFF
L1CD	1.661e^-3	1.985e^-3	1.381e^-3
F-Score	0.810	0.797	0.831

#	DC-DFFN	GFFM	LEFM	SDAFM	L1CD	F-Score
1	??				4.228e^-5	0.828
2	??	??			4.109e^-5	0.831
3	??	??	??		3.477e^-5	0.865
4	??	??		??	3.416e^-5	0.855
5	??	??	??	??	2.598e^-5	0.897