Real-Time Tracking of Smartwatch Orientation and Location by Multitask Learning

Arm posture tracking is essential for many applications, such as gesture recognition, fitness training, and motion-based controls. Smartwatches with Inertial Measurement Unit (IMU) sensors (i.e., accelerometer, gyroscope, and magnetometer) provide a convenient way to track the orientation and location of the wrist. Existing orientation estimations are based on predefined data fusion methods that do not consider the variations in the data quality of different IMU sensors. Existing location estimations rely on the estimated orientation results. A small orientation estimation error may cause high inaccuracy in location estimation. Moreover, these location estimation algorithms, e.g., Hidden Markov Model and Particle Filters, cannot provide real-time tracking on commercial mobile devices due to high computation overhead. This paper presents RTAT, a Real-Time Arm Tracking system that tackles the above limitations in a data-driven way. RTAT estimates both orientation and location simultaneously using a multitask learning neural network. It also incorporates a unique attention layer and a dedicated loss function to learn the dynamic relationship among IMU sensors. RTAT supports real-time tracking by performing model inference on smartphones. Finally, to train RTAT's neural network, we develop an easy-to-use labeled data collection system that uses a low-cost virtual reality system to provide orientation and location labels for the smartwatch. Extensive experiments show RTAT significantly outperforms existing state-of-the-art solutions in both accuracy and latency.