Augmented Reality-Based Multi-Cues Remote Industrial Collaboration System

doi:10.11871/jfdc.issn.2096-742X.2023.06.013

Abstract

Abstract:

[Objective] In order to solve the problems of casualties and property losses caused by low work efficiency and inadequate inspection in the traditional industrial inspection, this paper proposes a remote industrial collaboration system based on augmented reality technology. [Methods] The system enables remote experts and local workers to interact with rich bidirectional clues to complete remote collaboration tasks. In order to ensure that 2D clues drawn by experts and local workers in their respective scenes are mapped to 3D clues in real time, a diffusion virtual reality algorithm based on depth of camera focal length is adopted, which can obtain gesture rendering information in real time. And in order to verify the usability and advantages of the system, this paper simulates remote inspection tasks and equipment control tasks, and then designs comparative experiment and complete user surveys. [Results] The experimental results demonstrate the effectiveness of the augmented reality remote industrial collaboration system in reducing workers' workload. Additionally, it enhances the social presence of both remote parties and improves task efficiency compared to traditional audio and video conferences. Furthermore, this paper utilizesd the outcomes of usability tests to validate the authors' suggested ideas and implementesd improvements for future application scenarios. [Conclusions] Finally, this paper also studiesd the subjective feedback of user cognition in various forms, and puts forward suggestions to guide user behavior in industrial scenarios.

Key words: augmented reality, multiple clues, remote collaboration, visual analysis, human-computer interaction

ZHAO Qing, ZHANG Yue, ZHANG Chongbin, TIAN Dong, CUI Wenkai, SHAN Guihua. Augmented Reality-Based Multi-Cues Remote Industrial Collaboration System[J]. Frontiers of Data and Computing, 2023, 5(6): 138-152, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2023.06.013.

Figures/Tables 11

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

Fig.10

Fig.11

References 29

[1]	EDER M, HULLA M, MAST F, et al. On the application of Augmented Reality in a learning factory working environment[J/OL]. Procedia Manufacturing, 2020, 45: 7-12.
[2]	OBERMAIR F, ALTHALER J, SEILER U, et al. Maintenance with Augmented Reality Remote Support in Comparison to Paper-Based Instructions: Experiment and Analysis[C]// 2020 IEEE 7th International Conference on Industrial Engineering and Applications (ICIEA). Bangkok, Thailand: IEEE, 2020: 942-947.
[3]	MOURTZIS D, ZOGOPOULOS V, XANTHI F. Augmented reality application to support the assembly of highly customized products and to adapt to production re-scheduling[J]. The International Journal of Advanced Manufacturing Technology, 2019, 105(9): 3899-3910. doi: 10.1007/s00170-019-03941-6
[4]	MARRIOTT K, SCHREIBER F, DWYER T, et al. Immersive Analytics: 11190[M/OL]. Cham: Springer International Publishing, 2018[2022-09-08]. http://link.springer.com/10.1007/978-3-030-01388-2.
[5]	SERENO M, WANG X, BESANCON L, et al. Collaborative Work in Augmented Reality: A Survey[J]. IEEE Transactions on Visualization and Computer Graphics, 2020: 1-1.
[6]	KIM S, LEE G, SAKATA N, et al. Improving co-presence with augmented visual communication cues for sharing experience through video conference[C]// 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Munich, Germany: IEEE, 2014: 83-92.
[7]	Rasmussen T A, Huang W. SceneCam: Using AR to improve Multi-Camera Remote Collaboration[C]// SIGGRAPH Asia 2019 XR. 2019: 36-37.
[8]	JONES B, WITCRAFT A, BATEMAN S, et al. Mechanics of Camera Work in Mobile Video Collaboration[C]// Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. Seoul Republic of Korea: ACM, 2015: 957-966.
[9]	GAO L, BAI H, HE W, et al. Real-time visual representations for mobile mixed reality remote collaboration[C]// SIGGRAPH Asia 2018 Virtual & Augmented Reality on - SA '18. Tokyo, Japan: ACM Press, 2018: 1-2.
[10]	GAUGLITZ S, LEE C, TURK M, et al. Integrating the physical environment into mobile remote collaboration[C]// Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services - MobileHCI '12. San Francisco, California, USA: ACM Press, 2012: 241.
[11]	BAI H, SASIKUMAR P, YANG J, et al. A User Study on Mixed Reality Remote Collaboration with Eye Gaze and Hand Gesture Sharing[C]// Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Honolulu HI USA: ACM, 2020: 1-13.
[12]	GAO L, BAI H, LINDEMAN R, et al. Static local environment capturing and sharing for MR remote collaboration[C]// SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications on - SA '17. Bangkok, Thailand: ACM Press, 2017:1-6.
[13]	TEO T, LEE G A, BILLINGHURST M, et al. 360Drops: Mixed Reality Remote Collaboration using 360 Panoramas within the 3D Scene*[C]// SIGGRAPH Asia 2019 Emerging Technologies. Brisbane QLD Australia: ACM, 2019: 1-2.
[14]	TEO T, LAWRENCE L, LEE G A, et al. Mixed Reality Remote Collaboration Combining 360 Video and 3D Reconstruction[C/OL]// Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Glasgow Scotland Uk: ACM, 2019: 1-14[2022-08-30].
[15]	YANG J, SASIKUMAR P, BAI H, et al. The effects of spatial auditory and visual cues on mixed reality remote collaboration[J]. Journal on Multimodal User Interfaces, 2020, 14(4): 337-352. doi: 10.1007/s12193-020-00331-1
[16]	WANG P, BAI X, BILLINGHURST M, et al. 3DGAM: using 3D gesture and CAD models for training on mixed reality remote collaboration[J]. Multimedia Tools and Applications, 2021, 80(20): 31059-31084. doi: 10.1007/s11042-020-09731-7
[17]	CHANG Y C, WANG H C, CHU H kuo, et al. AlphaRead: Support Unambiguous Referencing in Remote Collaboration with Readable Object Annotation[C]// Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. Portland Oregon USA: ACM, 2017: 2246-2259.
[18]	THORAVI KUMARAVEL B, ANDERSON F, FITZMAURICE G, et al. Loki: Facilitating Remote Instruction of Physical Tasks Using Bi-Directional Mixed-Reality Telepresence[C]// Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. New Orleans LA USA: ACM, 2019: 161-174.
[19]	FUSSELL S R, SETLOCK L D, YANG J, et al. Gestures Over Video Streams to Support Remote Collaboration on Physical Tasks[J]. Human-Computer Interaction, 2004, 19(3): 273-309. doi: 10.1207/s15327051hci1903_3
[20]	TECCHIA F, ALEM L, HUANG W. 3D helping hands: a gesture based MR system for remote collaboration[C]// Proceedings of the 11th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry - VRCAI'12. Singapore, Singapore: ACM Press, 2012: 323.
[21]	BENKO H, ISHAK E W, FEINER S. Collaborative Mixed Reality Visualization of an Archaeological Excavation[C]// Third IEEE and ACM International Symposium on Mixed and Augmented Reality. Arlington, VA, USA: IEEE, 2004: 132-140.
[22]	JANG S, STUERZLINGER W, AMBIKE S, et al. Modeling Cumulative Arm Fatigue in Mid-Air Interaction based on Perceived Exertion and Kinetics of Arm Motion[C]// Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. Denver Colorado USA: ACM, 2017: 3328-3339.
[23]	Piumsomboon T, Dey A, Ens B, et al. The effects of sharing awareness cues in collaborative mixed reality[J]. Frontiers in Robotics and AI, 2019, 6: 5. doi: 10.3389/frobt.2019.00005 pmid: 33501022
[24]	HIGUCH K, YONETANI R, SATO Y. Can Eye Help You?: Effects of Visualizing Eye Fixations on Remote Collaboration Scenarios for Physical Tasks[C]// Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. San Jose California USA: ACM, 2016: 5180-5190.
[25]	PIUMSOMBOON T, DAY A, ENS B, et al. Exploring enhancements for remote mixed reality collaboration[C]// SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications on - SA '17. Bangkok, Thailand: ACM Press, 2017: 1-5.
[26]	Yuan Liao, Zhonghua Wang and Yanfen Luo. The design and implementation of a WebRTC based online video teaching system[C]// 2016 IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), Xi'an, China, 2016, 137-140.
[27]	LORETO S, ROMANO S P. Real-time communications in the web: Issues, achievements, and ongoing standardization efforts[J]. IEEE Internet Computing, 2012, 16(5): 68-73.
[28]	HARMS C, BIOCCA F. Internal consistency and reliability of the networked minds measure of social presence[C]// Seventh annual international workshop: Presence. Universidad Politecnica de Valencia Valencia, Spain, 2004: 246-251.
[29]	HART S G, STAVELAND L E. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research[M]// Advances in Psychology: 52. Elsevier, 1988: 139-183.