Egocentric Reconstruction of Human Bodies for Real-time Mobile Telepresence

A mobile 3D acquisition system has the potential to make telepresence significantly more convenient, available to users anywhere, anytime, without relying on any instrumented environments. Such a system can be implemented using egocentric reconstruction methods, which rely only on wearable sensors, such as head-worn cameras and body-worn inertial measurement units. Prior egocentric reconstruction methods suffer from incomplete body visibility as well as insufficient sensor data. This dissertation investigates an egocentric 3D capture system relying only on sensors embedded in commonly worn items such as eyeglasses, wristwatches, and shoes. It introduces three advances in egocentric reconstruction of human bodies. (1) A parametric-model-based reconstruction method that overcomes incomplete body surface visibility by estimating the user's body pose and facial expression, and using the results to re-target a high-fidelity pre-scanned model of the user. (2) A learning-based visual-inertial body motion reconstruction system that relies only on eyeglasses-mounted cameras and a few body-worn inertial sensors. This approach overcomes the challenges of self-occlusion and outside-of-camera motions, and allows for unobtrusive real-time 3D capture of the user. (3) A physically plausible reconstruction method based on rigid body dynamics, which reduces motion jitter and prevents interpenetrations between the reconstructed user's model and the objects in the environment such as the ground, walls, and furniture. This dissertation includes experimental results demonstrating the real-time, mobile reconstruction of human bodies in indoor and outdoor scenes, relying only on wearable sensors embedded in commonly-worn objects and overcoming the sparse observation challenges of egocentric reconstruction. The potential usefulness of this approach is demonstrated in a telepresence scenario featuring physical therapy training. ; Doctor of Philosophy

University of North Carolina at Chapel Hill Graduate School

2021

Dissertation ; [Doctoral and postdoctoral thesis]

English

Computer Graphics ; Virtual Reality ; Motion Capture ; Computer science ; Augmented Reality ; 3D Reconstruction ; Computer Vision

http://rightsstatements.org/vocab/InC-EDU/1.0/

https://doi.org/10.17615/bwp7-pq55
https://cdr.lib.unc.edu/downloads/ww72bm624?file=thumbnail
https://cdr.lib.unc.edu/downloads/ww72bm624

Carolina Digital Repository (UNC - University of North Carolina)  

• URL: https://cdr.lib.unc.edu/
• Continent: North America
• Country: us
• Number of documents: 82,417
• Open Access: 1,162 (2%)
• Type: Academic publications
• System: Fedora
• Content provider indexed in BASE since: 2015-09-04
• BASE URL: https://www.base-search.net/Search/Results?q=coll:ftcarolinadr