Handwriting-Assistant: Reconstructing Continuous Strokes with Millimeter-level Accuracy via Attachable Inertial Sensors

Pen-based handwriting has become one of the major human-computer interaction methods. Traditional approaches either require writing on the specific supporting device like the touch screen, or limit the way of using the pen to pure rotation or translation. In this paper, we propose Handwriting-Assistant, to capture the free handwriting of ordinary pens on regular planes with mm-level accuracy. By attaching the inertial measurement unit (IMU) to the pen tail, we can infer the handwriting on the notebook, blackboard or other planes. Particularly, we build a generalized writing model to correlate the rotation and translation of IMU with the tip displacement comprehensively, thereby we can infer the tip trace accurately. Further, to display the effective handwriting during the continuous writing process, we leverage the principal component analysis (PCA) based method to detect the candidate writing plane, and then exploit the distance variation of each segment relative to the plane to distinguish on-plane strokes. Moreover, our solution can apply to other rigid bodies, enabling smart devices embedded with IMUs to act as handwriting tools. Experiment results show that our approach can capture the handwriting with high accuracy, e.g., the average tracking error is 1.84mm for letters with the size of about 2cmx1cm, and the average character recognition rate of recovered single letters achieves 98.2% accuracy of the ground-truth recorded by touch screen.