Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition

Reviewer: Fernando Santos Osorio

Huang et al. present "a new approach for acquiring high-fidelity [and realistic] 3D facial performances" (motion capture of facial expressions and facial movements). According to the authors, their approach "leverages state-of-the-art motion capture technology and advanced 3D scanning technology for facial performance acquisition," reducing the time and effort required for 3D facial acquisition. They propose a hybrid system combining high spatial resolution face scans (using a laser scanner) and high temporal resolution motion capture data (using a marker-based motion capture system). In their experiments, they use a 12-camera Vicon motion capture system to record high-resolution dynamic facial movements, placing about 100 retro-reflective markers on the face and setting the acquisition rate to 240 frames per second. They also use a Minolta VIVID 910 laser scanner to record the high-resolution static facial geometry of an actor, which acquires a face mesh with 100k to 200k vertices in about 2.5 seconds and achieves an accuracy of 0.008 mm. The combination of both marker-based face motion capture and high-resolution static facial data enables the capture of high-quality facial expressions. The method is based on the following steps. First, the authors record facial performances of an actor using optical motion capture systems. Second, they perform facial analysis on the captured data and obtain a minimal set of face scans required for accurate facial reconstruction. Third, they register motion-capture markers to all of the face scans and build consistent dense surface correspondences across all of the face scans. Finally, they combine motion-capture data with the minimal set of the face scans to reconstruct high-fidelity facial performances in the blendshape interpolation framework. The facial data analysis (step two) automatically selects a minimum set of facial expressions by minimizing the reconstruction errors associated with the selected facial expressions. Then, the marker mesh registration (step three) is applied to register motion capture data between the two capturing systems. The authors also propose the use of a region-based fine-scale mesh registration process. The final results, presented in the tests conducted by the authors, acquire 3D facial performances of three subjects and demonstrate that the system can capture realistic dynamic wrinkles and fine-scale facial details with good performance. The results are best seen in the accompanying video of this paper (as the authors indicate). Online Computing Reviews Service