Nothing Special   »   [go: up one dir, main page]

skip to main content
research-article

Enabling Building Information Model-driven human-robot collaborative construction workflows with closed-loop digital twins

Published: 18 November 2024 Publication History

Abstract

The introduction of assistive construction robots can significantly alleviate physical demands on construction workers while enhancing both the productivity and safety of construction projects. Leveraging a Building Information Model (BIM) offers a natural and promising approach to driving robotic construction workflows. However, because of uncertainties inherent in construction sites, such as discrepancies between the as-designed and as-built components, robots cannot solely rely on a BIM to plan and perform field construction work. Human workers are adept at improvising alternative plans with their creativity and experience and thus can assist robots in overcoming uncertainties and performing construction work successfully. In such scenarios, it is critical to continuously update the BIM as work processes unfold so that it includes as-built information for the ensuing construction and maintenance tasks. This research introduces an interactive closed-loop digital twin framework that integrates a BIM into human-robot collaborative construction workflows. The robot’s functions are primarily driven by the BIM, but it adaptively adjusts its plans based on actual site conditions, while the human co-worker oversees and supervises the process. When necessary, the human co-worker intervenes in the robot’s plan by changing the task sequence or workspace geometry or requesting a new motion plan to help the robot overcome the encountered uncertainties. A drywall installation case study is conducted to verify the proposed workflow. In addition, experiments are carried out to evaluate the system performance using an industrial robotic arm in a research laboratory setting that mimics a construction site and in the Gazebo simulation. Integrating the flexibility of human workers and the autonomy and accuracy afforded by the BIM, the proposed framework offers significant promise of increasing the robustness of construction robots in the performance of field construction work.

Highlights

A BIM framework to support human-robot collaborative construction.
Technical and physical setup solutions from the preparation stage to end-of-work.
Automatic generation of interactive digital twin from the BIM and construction site.
Adjustment of work plan according to design-built deviations in field construction.
Demonstration of system performance in drywall installation and block placement.

References

[1]
F.H. Abanda, J.H.M. Tah, F.K.T. Cheung, BIM in off-site manufacturing for buildings, J. Build. Eng. 14 (2017) 89–102,.
[2]
P. Adami, P.B. Rodrigues, P.J. Woods, B. Becerik-Gerber, L. Soibelman, Y. Copur-Gencturk, G. Lucas, Effectiveness of VR-based training on improving construction workers’ knowledge, skills, and safety behavior in robotic teleoperation, Adv. Eng. Inform. 50 (2021),.
[3]
A. Adel, Computational Design for Cooperative Robotic Assembly of Nonstandard Timber Frame Buildings, Doctoral Thesis, ETH Zurich, 2020. 〈https://doi.org/10.3929/ethz-b-000439443〉.
[4]
A. Adel, A. Thoma, M. Helmreich, F. Gramazio, M. Kohler, Design of Robotically Fabricated Timber Frame Structures, in: Mexico City, Mexico, 2018: pp. 394–403. 〈https://doi.org/10.52842/conf.acadia.2018.394〉.
[5]
AGC, Construction Outlook National Survey Results, 2022, Associated General Contractors of America, 2022, 〈https://www.agc.org/sites/default/files/users/user21902/2022_Outlook_National_0.pdf〉 (Accessed 31 August 2023).
[6]
R. Aulin, M. Jingmond, Issues confronting women participation in the construction industry, in: Makere University, Uganda, 2011: pp. 312–318. 〈http://lup.lub.lu.se/record/1834659〉 (Accessed 31 August 2023).
[7]
Gabriel I. Barbash, Sherry A. Glied, New technology and health care costs — the case of robot-assisted surgery, N. Engl. J. Med. 363 (2010) 701–704,.
[9]
C. Brosque, J.T. Hawkins, T. Dong, J. Örn, M. Fischer, Comparison of on-site and off-site robot solutions to the traditional framing and drywall installation tasks, Constr. Robot 7 (2023) 19–39,.
[10]
J. Cai, A. Du, X. Liang, S. Li, Prediction-based path planning for safe and efficient human–robot collaboration in construction via deep reinforcement learning, J. Comput. Civ. Eng. 37 (2023),.
[11]
S. Chitta, I. Sucan, S. Cousins, MoveIt! [ROS Topics], IEEE Robot. Autom. Mag. 19 (2012) 18–19,.
[12]
P. Chotiprayanakul, D.K. Liu, G. Dissanayake, Human–robot–environment interaction interface for robotic grit-blasting of complex steel bridges, Autom. Constr. 27 (2012) 11–23,.
[13]
[14]
F. Correa, Robot-Oriented Design for Production in the context of Building Information Modeling, in: Auburn, AL, USA, 2016. 〈https://doi.org/10.22260/ISARC2016/0103〉.
[15]
F. Correa, Simulating Wood-framing Wall Panel’s Production with Timed Coloured Petri Nets, in: ISARC. Proceedings of the International Symposium on Automation and Robotics in Construction, IAARC Publications, Waterloo, Canada, 2019, pp. 1026–1033,.
[16]
C. Crick, G. Jay, S. Osentoski, B. Pitzer, O.C. Jenkins, Rosbridge: ROS for Non-ROS Users, in: H.I. Christensen, O. Khatib (Eds.), Robotics Research: The 15th International Symposium ISRR, Springer International Publishing, Cham, 2017, pp. 493–504,.
[17]
K. Daniilidis, Hand-eye calibration using dual quaternions, Int. J. Robot. Res. 18 (1999) 286–298,.
[18]
A. Davids, Urban search and rescue robots: from tragedy to technology, IEEE Intell. Syst. 17 (2002) 81–83,.
[19]
J.M. Davila Delgado, L. Oyedele, A. Ajayi, L. Akanbi, O. Akinade, M. Bilal, H. Owolabi, Robotics and automated systems in construction: understanding industry-specific challenges for adoption, J. Build. Eng. 26 (2019),.
[20]
O. Davtalab, A. Kazemian, B. Khoshnevis, Perspectives on a BIM-integrated software platform for robotic construction through Contour Crafting, Autom. Constr. 89 (2018) 13–23,.
[21]
M. Dawod, S. Hanna, BIM-assisted object recognition for the on-site autonomous robotic assembly of discrete structures, Constr. Robot 3 (2019) 69–81,.
[22]
F. Dembski, U. Wössner, M. Letzgus, M. Ruddat, C. Yamu, Urban digital twins for smart cities and citizens: the case study of Herrenberg, Germany, Sustainability 12 (2020) 2307,.
[23]
M. Deng, C.C. Menassa, V.R. Kamat, From BIM to digital twins: a systematic review of the evolution of intelligent building representations in the AEC-FM industry, ITcon 26 (2021) 58–83,.
[24]
P. Devadass, S. Stumm, S. Brell-Cokcan, Adaptive Haptically Informed Assembly with Mobile Robots in Unstructured Environments, in: Banff, AB, Canada, 2019. 〈https://doi.org/10.22260/ISARC2019/0063〉.
[25]
L. Ding, W. Jiang, Y. Zhou, C. Zhou, S. Liu, BIM-based task-level planning for robotic brick assembly through image-based 3D modeling, Adv. Eng. Inform. 43 (2020),.
[26]
G. Du, K. Wang, S. Lian, K. Zhao, Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review, Artif. Intell. Rev. 54 (2021) 1677–1734,.
[27]
Dusty, Dusty Robotics | Construction robots | BIM-driven Layout, (n.d.). 〈https://www.dustyrobotics.com/〉.
[28]
N. Emaminejad, R. Akhavian, Trustworthy AI and robotics: implications for the AEC industry, Autom. Constr. 139 (2022),.
[29]
E. Escamilla, M. Ostadalimakhmalbaf, B.F. Bigelow, Factors impacting hispanic high school students and how to best reach them for the careers in the construction industry, Int. J. Constr. Educ. Res. 12 (2016) 82–98,.
[30]
J. Fan, P. Zheng, S. Li, Vision-based holistic scene understanding towards proactive human–robot collaboration, Robot. Comput. -Integr. Manuf. 75 (2022),.
[31]
Y. Fang, Y.K. Cho, J. Chen, A framework for real-time pro-active safety assistance for mobile crane lifting operations, Autom. Constr. 72 (2016) 367–379,.
[32]
C. Feng, Camera Marker Networks for Pose Estimation and Scene Understanding in Construction Automation and Robotics., Thesis, 2015. 〈http://deepblue.lib.umich.edu/handle/2027.42/113481〉 (Accessed 23 April 2024).
[33]
C. Feng, V.R. Kamat, Plane registration leveraged by global constraints for context-aware AEC applications, Comput. Aided Civ. Eng. 28 (2013) 325–343,.
[34]
C. Feng, Y. Xiao, A. Willette, W. McGee, V.R. Kamat, Vision guided autonomous robotic assembly and as-built scanning on unstructured construction sites, Autom. Constr. 59 (2015) 128–138,.
[35]
C. Follini, V. Magnago, K. Freitag, M. Terzer, C. Marcher, M. Riedl, A. Giusti, D.T. Matt, BIM-integrated collaborative robotics for application in building construction and maintenance, Robotics 10 (2020) 2,.
[36]
B. Fu, W. Smith, D.M. Rizzo, M. Castanier, M. Ghaffari, K. Barton, Robust task scheduling for heterogeneous robot teams under capability uncertainty, IEEE Trans. Robot. 39 (2023) 1087–1105,.
[37]
A. Golabchi, M. Akula, V.R. Kamat, Leveraging BIM for Automated Fault Detection in Operational Buildings, in: Montreal, Canada, 2013. 〈https://doi.org/10.22260/ISARC2013/0020〉.
[38]
[39]
A. Hentout, M. Aouache, A. Maoudj, I. Akli, Human–robot interaction in industrial collaborative robotics: a literature review of the decade 2008–2017, Adv. Robot. 33 (2019) 764–799,.
[40]
14:00-17:00, ISO 10218-1:2011, ISO (n.d.). 〈https://www.iso.org/standard/51330.html〉 (accessed April 24, 2024).
[41]
V.R. Kamat, J.C. Martinez, 3D visualization of simulated construction operations, in: 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165), IEEE, Orlando, FL, USA, 2000, pp. 1933–1937,.
[42]
V.R. Kamat, J.C. Martinez, Automated generation of dynamic, operations level virtual construction scenarios, J. Inf. Technol. Constr. (ITcon) 8 (2003) 65–84.
[43]
S. Kim, S. Chang, D. Castro-Lacouture, Dynamic modeling for analyzing impacts of skilled labor shortage on construction project management, J. Manag. Eng. 36 (2020),.
[44]
D. Kim, J. Kim, K. Lee, C. Park, J. Song, D. Kang, Excavator tele-operation system using a human arm, Autom. Constr. 18 (2009) 173–182,.
[45]
K. Kim, J. Park, C. Cho, Framework for automated generation of constructible steel erection sequences using structural information of static indeterminacy variation in BIM, KSCE J. Civ. Eng. 24 (2020) 3169–3178,.
[46]
S. Kim, M. Peavy, P.-C. Huang, K. Kim, Development of BIM-integrated construction robot task planning and simulation system, Autom. Constr. 127 (2021),.
[47]
M. Krogius, A. Haggenmiller, E. Olson, Flexible Layouts for Fiducial Tags, IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS) 2019 (2019) 1898–1903,.
[48]
S. Lee, J.I. Moon, Introduction of Human-Robot Cooperation Technologyat Construction Sites, in: Sydney, Australia, 2014. 〈https://doi.org/10.22260/ISARC2014/0134〉.
[49]
S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, D. Quillen, Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection, Int. J. Robot. Res. 37 (2018) 421–436,.
[50]
R. Li, Z. Zou, Enhancing construction robot learning for collaborative and long-horizon tasks using generative adversarial imitation learning, Adv. Eng. Inform. 58 (2023),.
[51]
C.-J. Liang, V.R. Kamat, C.C. Menassa, W. McGee, Trajectory-based skill learning for overhead construction robots using generalized cylinders with orientation, J. Comput. Civ. Eng. 36 (2022),.
[52]
C.-J. Liang, W. McGee, C.C. Menassa, V.R. Kamat, Real-time state synchronization between physical construction robots and process-level digital twins, Constr. Robot 6 (2022) 57–73,.
[53]
C.-J. Liang, W. McGee, C.C. Menassa, V.R. Kamat, Real-time state synchronization between physical construction robots and process-level digital twins, Constr. Robot 6 (2022) 57–73,.
[54]
C.-J. Liang, X. Wang, V.R. Kamat, C.C. Menassa, Human–robot collaboration in construction: classification and research trends, J. Constr. Eng. Manag. 147 (2021),.
[55]
Y. Liu, M. Habibnezhad, H. Jebelli, Brain-computer interface for hands-free teleoperation of construction robots, Autom. Constr. 123 (2021),.
[56]
K.M. Lundeen, S. Dong, N. Fredricks, M. Akula, J. Seo, V.R. Kamat, Optical marker-based end effector pose estimation for articulated excavators, Autom. Constr. 65 (2016) 51–64,.
[57]
K.M. Lundeen, V.R. Kamat, C.C. Menassa, W. McGee, Scene understanding for adaptive manipulation in robotized construction work, Autom. Constr. 82 (2017) 16–30,.
[58]
K.M. Lundeen, V.R. Kamat, C.C. Menassa, W. McGee, Autonomous motion planning and task execution in geometrically adaptive robotized construction work, Autom. Constr. 100 (2019) 24–45,.
[59]
A.A. Malik, A. Brem, Digital twins for collaborative robots: a case study in human-robot interaction, Robot. Comput. -Integr. Manuf. 68 (2021),.
[60]
B.R.K. Mantha, M.K. Jung, B. García De Soto, C.C. Menassa, V.R. Kamat, Generalized task allocation and route planning for robots with multiple depots in indoor building environments, Autom. Constr. 119 (2020),.
[61]
A. McClymonds, S. Asadi, A. Wagner, R.M. Leicht, Information Exchange for Supporting BIM to Robotic Construction, in: Construction Research Congress 2022, American Society of Civil Engineers, Arlington, Virginia, 2022, pp. 839–848,.
[62]
S. Meschini, Novel applications offered by integration of robotic tools in BIM-based design workflow for automation in construction processes, in: 2016. 〈https://mediatum.ub.tum.de/1484218〉 (Accessed 4 April 2024).
[63]
NIBS, National BIM Standard-United States® Version 3 | National BIM Standard - United States, (2015). 〈https://www.nationalbimstandard.org/nbims-us〉 (Accessed 22 March 2022).
[64]
J. Park, Y.K. Cho, D. Martinez, A BIM and UWB integrated mobile robot navigation system for indoor position tracking applications, J. Constr. Eng. Proj. Manag. 6 (2016) 30–39,.
[65]
J. Park, Y.K. Cho, Point cloud information modeling: deep learning–based automated information modeling framework for point cloud data, J. Constr. Eng. Manag. 148 (2022),.
[66]
S. Park, X. Wang, C.C. Menassa, V.R. Kamat, J.Y. Chai, Natural language instructions for intuitive human interaction with robotic assistants in field construction work, Autom. Constr. 161 (2024),.
[67]
S. Park, H. Yu, C.C. Menassa, V.R. Kamat, A comprehensive evaluation of factors influencing acceptance of robotic assistants in field construction work, J. Manag. Eng. 39 (2023),.
[68]
L. Pérez, E. Diez, R. Usamentiaga, D.F. García, Industrial robot control and operator training using virtual reality interfaces, Comput. Ind. 109 (2019) 114–120,.
[69]
R. Pietroforte, Shop drawing process of stone veneered cladding systems, J. Archit. Eng. 3 (1997) 70–79,.
[70]
K. Qian, A. Song, J. Bao, H. Zhang, Small teleoperated robot for nuclear radiation and chemical leak detection, Int. J. Adv. Robot. Syst. 9 (2012) 70,.
[71]
R.M.& Associates, Rhino 7, 〈Www.Rhino3d.Com〉 (n.d.). 〈https://www.rhino3d.com/7/〉 (accessed April 23, 2024).
[72]
Robert McNeel & Associates, Rhino.Inside, 〈Www.Rhino3d.Com〉 (2024). 〈https://www.rhino3d.com/features/rhino-inside/〉 (Accessed 11 January 2024).
[73]
P.B. Rodrigues, R. Singh, M. Oytun, P. Adami, P.J. Woods, B. Becerik-Gerber, L. Soibelman, Y. Copur-Gencturk, G.M. Lucas, A multidimensional taxonomy for human-robot interaction in construction, Autom. Constr. 150 (2023),.
[74]
J.J. Roldán, E. Peña-Tapia, D. Garzón-Ramos, J. de León, M. Garzón, J. del Cerro, A. Barrientos, Multi-robot Systems, Virtual Reality and ROS: Developing a New Generation of Operator Interfaces, in: A. Koubaa (Ed.), Robot Operating System (ROS), Springer International Publishing, Cham, 2019, pp. 29–64,.
[75]
S. Sharif, T.R. Gentry, L.M. Sweet, Human-Robot Collaboration for Creative and Integrated Design and Fabrication Processes, in: Auburn, AL, USA, 2016. 〈https://doi.org/10.22260/ISARC2016/0072〉.
[76]
L. She, Y. Cheng, J.Y. Chai, Y. Jia, S. Yang, N. Xi, Teaching Robots New Actions through Natural Language Instructions, in: The 23rd IEEE International Symposium on Robot and Human Interactive Communication, IEEE, Edinburgh, UK, 2014, pp. 868–873,.
[78]
A. Suresh, D. Gaba, S. Bhambri, D. Laha, Intelligent Multi-fingered Dexterous Hand Using Virtual Reality (VR) and Robot Operating System (ROS), in: J.-H. Kim, H. Myung, J. Kim, W. Xu, E.T. Matson, J.-W. Jung, H.-L. Choi (Eds.), Robot Intelligence Technology and Applications 5, Springer International Publishing, Cham, 2019, pp. 459–474,.
[79]
J. Teizer, A. Blickle, T. King, O. Leitzbach, D. Guenther, H. Mattern, M. König, BIM for 3D Printing in Construction, in: A. Borrmann, M. König, C. Koch, J. Beetz (Eds.), Building Information Modeling, Springer International Publishing, Cham, 2018, pp. 421–446,.
[80]
P. Tsarouchi, A.-S. Matthaiakis, S. Makris, G. Chryssolouris, On a human-robot collaboration in an assembly cell, Int. J. Comput. Integr. Manuf. 30 (2017) 580–589,.
[81]
R. Volk, J. Stengel, F. Schultmann, Building Information Modeling (BIM) for existing buildings — literature review and future needs, Autom. Constr. 38 (2014) 109–127,.
[82]
X. Wang, C.-J. Liang, C. Menassa, V. Kamat, Real-Time Process-Level Digital Twin for Collaborative Human-Robot Construction Work, in: Kitakyushu, Japan, 2020. 〈https://doi.org/10.22260/ISARC2020/0212〉.
[83]
X. Wang, C.-J. Liang, C.C. Menassa, V.R. Kamat, Interactive and immersive process-level digital twin for collaborative human–robot construction work, J. Comput. Civ. Eng. 35 (2021),.
[84]
X. Wang, S. Wang, C.C. Menassa, V.R. Kamat, W. McGee, Automatic high-level motion sequencing methods for enabling multi-tasking construction robots, Autom. Constr. 155 (2023),.
[85]
O. Wong Chong, J. Zhang, R.M. Voyles, B.-C. Min, BIM-based simulation of construction robotics in the assembly process of wood frames, Autom. Constr. 137 (2022),.
[86]
L. Xu, C. Feng, V.R. Kamat, C.C. Menassa, An Occupancy Grid Mapping enhanced visual SLAM for real-time locating applications in indoor GPS-denied environments, Autom. Constr. 104 (2019) 230–245,.
[87]
S. Yoon, Y. Kim, M. Park, C.R. Ahn, Effects of spatial characteristics on the human–robot communication using deictic gesture in construction, J. Constr. Eng. Manag. 149 (2023),.
[88]
H. Yu, V.R. Kamat, C.C. Menassa, Cloud-based hierarchical imitation learning for scalable transfer of construction skills from human workers to assisting robots, J. Comput. Civ. Eng. 38 (2024),.
[89]
H. Yu, V.R. Kamat, C.C. Menassa, W. McGee, Y. Guo, H. Lee, Mutual physical state-aware object handover in full-contact collaborative human-robot construction work, Autom. Constr. 150 (2023),.
[90]
R. Zhang, S. Lee, M. Hwang, A. Hiranaka, C. Wang, W. Ai, J.J.R. Tan, S. Gupta, Y. Hao, G. Levine, R. Gao, A. Norcia, L. Fei-Fei, J. Wu, NOIR: Neural Signal Operated Intelligent Robots for Everyday Activities, (2023). 〈http://arxiv.org/abs/2311.01454〉 (accessed April 4, 2024).
[91]
Z. Zhang, K.-Y. Lin, J.-H. Lin, 2SAFE: a health belief model-integrated framework for participatory ergonomics, Theor. Issues Ergon. Sci. 24 (2023) 281–298,.
[92]
J. Zhang, H. Luo, J. Xu, Towards fully BIM-enabled building automation and robotics: a perspective of lifecycle information flow, Comput. Ind. 135 (2022),.
[93]
T. Zhou, Q. Zhu, J. Du, Intuitive robot teleoperation for civil engineering operations with virtual reality and deep learning scene reconstruction, Adv. Eng. Inform. 46 (2020),.
[94]
A. Zhu, P. Pauwels, B. De Vries, Smart component-oriented method of construction robot coordination for prefabricated housing, Autom. Constr. 129 (2021),.
[95]
Q. Zhu, T. Zhou, J. Du, Upper-body haptic system for snake robot teleoperation in pipelines, Adv. Eng. Inform. 51 (2022),.

Index Terms

  1. Enabling Building Information Model-driven human-robot collaborative construction workflows with closed-loop digital twins
      Index terms have been assigned to the content through auto-classification.

      Recommendations

      Comments

      Please enable JavaScript to view thecomments powered by Disqus.

      Information & Contributors

      Information

      Published In

      cover image Computers in Industry
      Computers in Industry  Volume 161, Issue C
      Oct 2024
      247 pages

      Publisher

      Elsevier Science Publishers B. V.

      Netherlands

      Publication History

      Published: 18 November 2024

      Author Tags

      1. Construction robotics
      2. Building Information Model
      3. Digital twin
      4. Human-robot collaboration

      Qualifiers

      • Research-article

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • 0
        Total Citations
      • 0
        Total Downloads
      • Downloads (Last 12 months)0
      • Downloads (Last 6 weeks)0
      Reflects downloads up to 18 Nov 2024

      Other Metrics

      Citations

      View Options

      View options

      Login options

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media