IMUPoser: Full-Body Pose Estimation using IMUs in Phones, Watches, and Earbuds

There exists a wide range of approaches and solutions to estimate users’ pose using external sensors. Commercial systems such as Vicon [58] and OptiTrack [41] use specialized hardware, including high-speed infrared cameras that track retroreflective markers attached to users’ whole body or individual parts, such as the face or hands. After a calibration procedure, these systems can track large spaces at millimeter accuracy. These approaches are often used for movies, games, and character animation. The cost and setup requirements, however, preclude most consumer applications.

Current approaches for whole-body pose estimation in consumer applications typically rely on cheaper sensors and require less calibration. Depth cameras such as the Microsoft Kinect [45, 53, 61] and Intel RealSense [22] provide sufficient pose accuracy using medium-cost sensors for applications in VR and gaming. Zimmermann et al. [71] and Michael et al. [37], for example, extend these approaches and combine depth imagery with RGB data for improved accuracy. Such commercially-available sensors provide a good balance between cost, availability and accuracy, however, they are generally immobile setups. Recent successes in computer vision and deep learning have made it possible to extract pose data from monocular RGB cameras. This includes approaches that infer 2D pose of one or multiple humans from a single image [11, 12, 46], multiple cameras [14], or estimating 3D poses from 2D images [34].

There also exists specialized external hardware for pose tracking in VR and AR [33]. For example, the HTC Vive [19, 20], Oculus Rift [36] and PlayStation VR [54] track the head, hand controllers and other limb-borne accessories using external sensor base stations. Any un-sensed joints can be roughly estimated with inverse kinematics [47, 50]. Other non-worn, external approaches for pose estimation include capacitive sensing [68], magnetic fields [44, 49], RF [69], and mechanical linkages [55].

Pose estimation using body-worn sensors is more portable and flexible than systems requiring external components. Much research has focused on capturing specific body parts. For instance, hand pose is of great importance in VR/AR applications, and has been tracked using e.g., wrist-worn cameras [10, 29, 62], electrical impedance tomography [67], electromyography [30], depth sensors [16], magnetic trackers [13], and specialized gloves [17]. Faces are important too, most often captured using cameras [4, 57], but other methods such as ultrasound [23] and electromyography [18] have also been explored.

Our research is more concerned with whole-body pose estimation. Many body-worn sensing approaches exist, ranging from exoskeletons [35], ultrasonic beacons [59], pressure sensors [66] and RFID [27]. Body-worn camera approaches are particularly popular, such as work by Shiratori et al. [52], Ng et al. [42], and Ahuja et al. [2, 3]. All of the latter approaches require specialized additional hardware that most users do not own. The goal of our work is to bring the flexibility of body-worn pose estimation to users without requiring them to purchase any new devices. One prior work with a similar mantra is Pose-On-The-Go [6], which estimates a user’s full-body pose using only the sensors in a smartphone and when held in the hand. However, much of the full-body pose is guessed (rather than tracked) by measuring absolute movement and using an animated, rigged (IK) avatar.

In this section, we focus on pose systems relying exclusively on worn IMUs, which most closely matches our technical approach (Table 1 provides an overview of prior systems). While single IMUs have been used to track individual limbs (such as arm pose in ArmTrack [51]), it is more common to see "fleets" of IMUs distributed across the body (e.g., the popular XSens [63] suit) for full-body pose estimation. Importantly, these setups are homogeneous in terms of IMUs (and thus performance and noise) and tend to use high-quality sensors running at high framerates not typically seen in consumer mobile devices. As we found, the IMUs utilized in Apple’s own ecosystem vary by device, and as such, the data varies in quality, noise and framerate.

Among prior work using many body-worn IMUs, we see Tautges et al. [56] able to generate visually plausible motion streams using four XSens IMUs. Sparse Inertial Poser [60] and Deep Inertial Poser [21] use optimization and deep learning-based methods for full-body pose estimation using between 6 and 17 body-worn IMUs. Both systems use SMPL [31, 48], a statistical body model, as their pose output. Approaches such as TransPose [65] or Physical Inertial Poser [64] build on such efforts and provide more accurate representations and better models. All these works leverage the fact that the employed IMUs have known and calibrated positions, and the same noise profiles.

For the learning architecture, we use a two-layer Bidirectional LSTM, inspired by prior works [21, 65]. Although we did experiment with newer architectures such as transformers, we found these models did not perform well in practice. LSTMs produced smoother output predictions than the other models we tested. For each available IMU, our system uses orientation (represented as a 3 × 3 rotation matrix) as well as acceleration as input, both in a global coordinate frame of reference. In contrast to prior work, we do not normalize these inputs to be relative to a root IMU sensor location, such as the pelvis, as our available devices vary. We flatten and concatenate these inputs to form an input vector of size 60: 5 possible IMU locations × (3 acceleration axes + 3 × 3 orientation rotation matrix), which we input to the model. Of note, our model can ingest any subset of the available IMU data, with absent devices masked (i.e., values set to zero).

The input vector is first transformed into an embedding of dimension 256 using a ReLU [40] activated linear layer. Next, these embeddings are fed sequentially into a Bidirectional LSTM of hidden dimension 256. A final linear layer outputs 144 SMPL [31] pose parameters — 24 joints represented as 6D rotations (which is a smoother representation space [70]). SMPL also provides a body mesh (6890 vertices), which can be seen in Figures 1, 4 and 7. In total, our neural network model has 10.7M trainable parameters. During training, a forward kinematics module calculates joint positions from these pose parameters and further minimizes it with respect to the ground truth.

To train our pose model, we required a significant volume of data. For this, we can leverage existing motion capture databases to generate a large synthetic corpus. Specifically, we use AMASS [32], a compilation of 24 motion capture datasets (ACCAD, BioMotion, CMU MoCap, MIXAMO, Human Eva, Human 3.6M, etc.) totaling almost 63 hours of high-quality, high-resolution motion capture data in the SMPL [31] format. A wide variety of motions and activity contexts are included, such as locomotion, sports, dancing, exercising, cooking, and freestyle interactions. For additional details on the composition of the AMASS dataset, please refer to [32]. We note that AMASS has been used in much prior work [21, 64, 65] as the basis for deriving synthetic datasets.

The consumer devices we use for our study and real-time demo (described in Sections 5.2.1 and 8) run at a common framerate of 25 FPS. Thus we resample AMASS’ 60 ∼ 120 FPS data to 25 FPS. We then follow the synthetic data generation process used in TransPose [65] and DIP [21]. In short, we "attach" virtual IMUs to specific vertices in the SMPL mesh (the left and right wrists, the left and right front pant pockets, and the scalp) and compute synthetic acceleration data using adjacent frames in the global frame of reference. To generate synthetic orientation data, we calculate joint rotations relative to the global frame by compounding local rotations starting from the joint to the pelvis (root) following the SMPL kinematic chain. We scale acceleration data (m/s²) by 30 to be suitable for neural networks [65]. Finally, rather than adding synthetic high-frequency noise to our dataset, we instead smooth both synthetic and real-world data using an averaging window of length 5 frames (200 ms), similar to [26].

We use this pipeline to create 24 sets of data, one for each of our 24 device-location combinations (Figure 3), which we combine into a single dataset. We simulate missing devices by masking-out (i.e., zeroing-out) IMU data for those locations. For example, even in our best-case scenario of three devices present, this means that 2/5^ths of the input vector is null. 63 hours of AMASS data × 25 FPS × 24 device-location combinations yields 134.8M synthetic IMU instances with paired ground truth SMPL poses for training.

The model is trained end-to-end using PyTorch and PyTorch Lightning deep learning frameworks. We use a batch size of 256 and update the weights using the Adam optimizer with a learning rate of 3e^{− 4}. While training, we use non-overlapping windows of paired IMU and pose data in 5-second (125 samples) chunks. As mentioned earlier, we train our model to regress to full-body pose and full-body joint positions using mean squared error (MSE) loss. Our total loss is the sum of these two individual losses. We train our model for 80 epochs (22 hours) on an NVIDIA Titan X GPU.

As the last step of our inference pipeline, we adopt the Inverse Kinematic refinement method presented in [25] to perform a final refinement of our output pose. Although our model predicts the rotation of legs, hands and head, it does not necessarily fully honor the absolute orientation offered by the IMUs, even when weighted heavily in our loss term. However, it is logical to take advantage of IMU orientation for limbs with devices, as it is both an absolute value and considerably less noisy than accelerometer data. More specifically, as we have absolute orientation from the IMUs, we optimize certain bone orientations for each instrumented joint. In particular, for the wrist joint (smartwatch/phone), we optimize the elbow and the shoulder orientations, and similarly for the head (earbuds/phone) and hip (smartphone/earbuds case) joints. We implement this using the PyTorch framework and optimize this error using the MSE loss and the Adam optimizer. We allow this optimization to run for 10 iterations on each frame, which we found to not impede real-time performance.

To test the performance of our model on real (and not synthetic) IMU data, we use DIP-IMU [21], an IMU-based MoCap dataset. While smaller than the AMASS dataset we used for training, it offers a good variety of poses and activities across five classes: upper-body (arm raises, stretches, swings, etc.), lower-body (leg raises, squats, lunges, etc.), interaction (gestures to interact with everyday objects), freestyle (jumping jacks, punching, kicking, etc.) and locomotion (walking, side steps, etc.). A secondary benefit of using DIP-IMU is that it has been used for evaluation in other similar works [21, 26, 64, 65], permitting direct comparison. DIP-IMU used the commercially-available Xsens [63] IMU-based system to capture data from 10 participants. The data is sampled at 60 Hz, leading to a total dataset size of approximately 90 mins.

As noted above, DIP-IMU used the professional-grade XSens system for data collection, which costs approximately $4000 USD. All of the IMUs are matched, offering similar noise and tracking performance. To complement this dataset with a consumer device equivalent, we collected our own dataset.

In order to compare with prior works, we follow the exact method detailed in [21, 26, 64, 65]. Specifically, we use data from the first eight participants of DIP-IMU as training data, with the last two participants used for testing. We fine-tune our AMASS-trained model using this training data downsampled to 25 Hz to match both our AMASS training data and our real-time system’s capabilities. We further test this model on our IMUPoser Dataset, helping to assess real-world accuracy and performance. Our model is evaluated in an online fashion. In particular, we feed a rolling window of 125 samples (5-second history) with a 1-sample overlap, emulating real-world use. This data is smoothed using an averaging filter, as described in Section 4.2. We analyze these results using different evaluation metrics across various device-location combinations. Also following prior work [21, 64, 65], we make use of the following evaluation metrics to quantify the performance of our full-body pose estimation pipeline:

We use mesh error (MPJVE) as our primary evaluation metric for most tasks, due to its ease of understanding and its utility as a benchmark for comparison with prior work.

To simplify presentation of results, we group the 24 possible device-location combinations (Figure 3) into 12 supersets based on the number of devices present and their body locations (ignoring left/right placements). Figure 6 presents the results for the IMUPoser and DIP-IMU datasets. Note, that our model has not been fine-tuned on the IMUPoser Dataset. Across all device combinations, we find a MPJVE of 14.1 cm on the IMUPoser Dataset and a MPJVE of 12.1 cm with DIP-IMU. When averaging the results across both datasets, having one device on the user results in a MPJVE of 16.27 cm (SD=9.93 cm), which decreases to 13.9 cm (SD=8.36 cm) when a second device is present. The lowest error, unsurprisingly, is when three devices are present – a MPJVE of 11.1 cm (SD=6.51 cm) across all possible three-device combinations.

Figure 7 offers example mesh predictions across different device-location combinations. As expected, accurate head orientation estimation is only plausible when earbuds are present. Other times the head regresses to the most natural orientation given where the body is facing. Global body orientation works best when at least two devices are present. Lastly, motions that have a characteristic cadence, such as walking, work well across all combinations. Similarly, activities with symmetric limb motions, such as jumping jacks, work fairly well even with no sensor data from important limbs. On the other hand, activities with uncorrelated limb motions fail unless limbs are instrumented.

Figure 8 provides a breakdown of system accuracy across different body regions for the IMUPoser and DIP-IMU datasets. We note that the accuracy for a limb with an instrumented point is always greater than that of an uninstrumented one. For example, averaging across both datasets and with an IMU present on the right hand, the MPJVE is 14.65 cm for the right arm (right hand = 17.2 cm) vs. 21.6 cm for the left arm (left hand = 26.9 cm). Unsurprisingly, the highest error is when none of the limbs in a particular body region have IMU data.

Also unsurprising is that the lowest error is achieved when both left and right limbs have IMUs present. For example, with only one IMU on the arms, the MPJVE for both arms is 18 cm (right hand = 22.17 cm; left hand = 20.4 cm). Whereas with both arms having IMUs, the MPJVE is 14.5 cm (right hand = 17.35 cm; left hand = 16.9 cm). A partial exception to this trend is the legs. Unlike the arms, which can move independently, legs tend to move in tandem (out of phase when walking, or in phase for activities such as jumping). This means that even one IMU on the legs is still highly effective at predicting both legs, and two IMUs offer just a modest gain. Looking at our results, the MPJVE for the left leg is 10.3 cm (left foot = 15.65 cm) when the IMU is in the left pocket, and the error for the right leg is 10.4 cm (right foot = 16 cm) when the IMU is in the right pocket. When both IMUs are present (i.e., left and right pockets), the error of the left and right legs drop very modestly to 10.05 cm and 9.75 cm, respectively.

We note that error accumulates along the kinematic chain (see Figure 7). Across all conditions, the average error of the end-effectors (left hand, right hand, left foot, right foot, head) is 20.28 cm and 17.29 cm on the IMUPoser Dataset and DIP-IMU Dataset, respectively (vs. 12.92 cm and 11.23 cm for joints that are not end-effectors).

To the best of our knowledge, no prior research has investigated deriving full-body pose from such a sparse set of consumer-grade devices equipped with IMUs. Table 2 offers a quantitative comparison against key prior work, all evaluated on the same DIP-IMU Dataset [21].

Unsurprisingly, for a system that uses between 1 and 3 IMUs, our model is less accurate than those utilizing 6 sensors (i.e., IMUs placed on each limb). However, compared to DIP [21] and Transpose [65], our MPJVE is only worse by 3.2 cm and 5.0 cm, respectively. It is interesting to note that the Jitter of our system is in line with prior work (1.9 vs. 1.4 of TransPose). At a high level, even with an impoverished sensing configuration, we are able to produce natural, realistic and smooth pose estimation sequences. In the future, we hope to combine physics-backed models (as in PIP) to further improve the pose estimation of our system.

To determine where devices are located on the body, we require three pieces of information from the user, which we envision being collected when a user first purchases a device. 1) In which pocket do they typically store their phone? 2) In which hand do they typically hold their phone? And 3) On which arm would they wear a smartwatch? After this basic initialization, we use a series of automated heuristics.

We make the assumption a smartphone is held in the hand if the screen is on and the IMU is reporting even slight motion. If the user is wearing a smartwatch, we can use the distance between the watch and phone (provided by Apple’s NINearbyObject API [9], which uses UWB) to guess the holding hand automatically (see Figure 10). If no smartwatch is worn, our system falls back on the hand specified by the user during setup. If the smartphone screen is off and the IR proximity sensor is triggered, we assume the phone is in a pocket. If the user has a smartwatch, we can similarly use UWB-derived distance to guess the pocket. If no smartwatch is worn, we default to the user-specified pocket.

As most users wear watches in a consistent location, the logic for smartwatches is simpler. If it is connected to the iPhone and moving, we assume it is worn on the user-specified hand. Similarly, for Airpods, if they are connected to the iPhone, we know they are in the ear. When in their charging case, Airpods go to sleep and stop transmitting IMU data. However, we believe that Apple could modify the Airpods firmware such that in the future they could continue to transmit IMU data even when stored in a pocket.

As a preliminary evaluation of our active device tracking prediction, we ran a user study with 7 participants (5 identified as male, 2 identified as female; mean age 27.8; all right-handed with a preference for wearing watches on the left wrist). The study lasted approximately 15 minutes and paid $5. To initialize our system, we recorded participants’ answers to the three preference questions listed in the previous section. We then asked users to transition between 15 device combinations, in a random order, documented in Figure 9. When a device was not in a requested set, it was set aside on a nearby table. For each requested device-location combination, we asked participants to walk around for about 10 seconds, then sit down briefly, rise to stand again, and lastly return to the starting position. Before the next trial began, the necessary devices were given or taken from the participants. Throughout the study, our active device tracking process ran, making live predictions about what devices were active and where they were located. A trained experimenter conducted the study, marking the start and stop of each device combination trial, alongside ground truth labels.

Across all participants and all data instances, the accuracy of earbuds and smartwatch tracking was 100%, owing to their known locations and very reliable detection of worn vs. not worn. Smartphone tracking is the most challenging, with five possible states (not present, left pocket, right pocket, left hand, right hand). We found the instance-wise accuracy for smartphone tracking was 90.8%.

As a proof-of-concept implementation, we use an Apple iPhone 11 Pro, Apple Watch Series 6, and AirPods Pro. Apple offers a mature inter-device API that allows these devices to exchange data. Each device reports 6DOF IMU data at different rates, with the slowest being AirPods at roughly 25 FPS. We also note that although AirPods come as a pair, they fuse their individual IMUs into a single 6DOF head estimate.

As a proof of concept, we use an iPhone optionally connected to an Apple Watch and AirPods. The iPhone streams all available IMU data back to a MacBook Air (2021), which runs our active device tracking and pose estimation processes, with a mean inference time of 26.8 ms. We believe our model could be run on a mobile phone with additional engineering effort. Regardless of where the model runs, it is capped at 25 FPS, the reporting frequency of our slowest IMU (Airpods). Before running our system, we must perform the same calibration as in data collection (see Section 5.2.2). For real-time output, we visualize the SMPL mesh.