research-article

Using mobile phones to write in air

Authors:

Sandip Agrawal,

Ionut Constandache,

Shravan Gaonkar,

Romit Roy Choudhury,

Frank DeRuyterAuthors Info & Claims

MobiSys '11: Proceedings of the 9th international conference on Mobile systems, applications, and services

Pages 15 - 28

https://doi.org/10.1145/1999995.1999998

Published: 28 June 2011 Publication History

Abstract

Numerous sensors in modern mobile phones enable a range of people-centric applications. This paper envisions a system called PhonePoint Pen that uses the in-built accelerometer in mobile phones to recognize human writing. By holding the phone like a pen, a user should be able to write short messages or draw simple diagrams in the air. The acceleration due to hand gestures can be translated into geometric strokes, and recognized as characters. We prototype the PhonePoint Pen on the Nokia N95 platform, and evaluate it through real users. Results show that English characters can be identified with an average accuracy of 91.9%, if the users conform to a few reasonable constraints. Future work is focused on refining the prototype, with the goal of offering a new user-experience that complements keyboards and touch-screens.

References

[1]

AiLive LiveMove pro. AiLive Inc. http://www.ailive.net/liveMovePro.html.

[2]

Microsoft. Write in The Air, TechFest 2009. http://www.youtube.com/watch?v=WmiGtt0v9CE.

[3]

S. Agrawal, I. Constandache, S. Gaonkar, and R. Roy Choudhury. Phonepoint pen: using mobile phones to write in air. In ACM MobiHeld, 2009.

Digital Library

[4]

C. Alvarado and R. Davis. Sketchread: a multi-domain sketch recognition engine. In ACM UIST, 2004.

Digital Library

[5]

A. Arranz. Niime. http://www.niime.com/.

[6]

V. Balakrishnan and P. H. Yeow. Sms usage satisfaction: Influences of hand anthropometry and gender. In Human IT 9.2, 2007.

[7]

V. Balakrishnan and P. H. Yeow. A study of the effect of thumb sizes on mobile phone texting satisfaction. In Journal of Usability Studies, 2008.

[8]

T. Baudel and M. Beaudouin-Lafon. Charade: remote control of objects using free-hand gestures. Commun. ACM, 1993.

Digital Library

[9]

X. Cao and R. Balakrishnan. Visionwand: interaction techniques for large displays using a passive wand tracked in 3d. ACM Trans. Graph., 2004.

Digital Library

[10]

D. Goldberg and C. Richardson. Touch-typing with a stylus. In ACM CHI, 1993.

Digital Library

[11]

M. D. Gross. The electronic cocktail napkin--a computational environment for working with design diagrams. Design Studies, 1996.

[12]

K. Hinckley, P. Baudisch, G. Ramos, and F. Guimbretiere. Design and analysis of delimiters for selection-action pen gesture phrases in scriboli. In ACM CHI, 2005.

Digital Library

[13]

J. Kela, P. Korpipaa, J. Mantyjarvi, S. Kallio, G. Savino, L. Jozzo, and D. Marca. Accelerometer-based gesture control for a design environment. Personal Ubiquitous Comput., 2006.

Digital Library

[14]

J. Liu, Z. Wang, and L. Zhong. uWave: Accelerometer-based Personalized Gesture Recognition and Its Applications. Mar. 2009.

[15]

LiveScribe. Smartpen. http://www.livescribe.com/.

[16]

Logitech. Air mouse. http://www.logitech.com.

[17]

I. S. MacKenzie and S. X. Zhang. The immediate usability of graffiti. In Graphics Interface, 1997.

Digital Library

[18]

J. Mantyjarvi, J. Kela, P. Korpipaa, and S. Kallio. Enabling fast and effortless customisation in accelerometer based gesture interaction. In Proceedings of the 3rd international conference on Mobile and ubiquitous multimedia, 2004.

Digital Library

[19]

B. Milner. Handwriting recognition using acceleration-based motion detection. In Document Image Processing and Multimedia, IEEE Colloquium on, 1999.

[20]

Nintendo. Wii console. http://www.nintendo.com/wii.

[21]

Nintendo. Wii motion plus. http://www.nintendo.com/whatsnew.

[22]

Nokia. Virtual keyboard. http://www.unwiredview.com/wp-content/uploads/2008/01/nokia-virtual-keyboard%-patent.pdf.

[23]

PlayStation. Move. http://us.playstation.com/ps3/playstation-move/.

[24]

C. Soriano, G. K. Raikundalia, and J. Szajman. A usability study of short message service on middle-aged users. In OZCHI, 2005.

Digital Library

[25]

C. Sung-Do, L. A.S., and L. Soo-Young. On-line handwritten character recognition with 3d accelerometer. International Conference on Information Acquisition, 2006.

[26]

L. von Ahn, M. Blum, and J. Langford. Telling humans and computers apart automatically. Commun. ACM, 2004.

Digital Library

[27]

J. Wang and J. Canny. Tinymotion: camera phone based interaction methods. In CHI '06 extended abstracts on Human factors in computing systems, 2006.

Digital Library

[28]

L. Williams. Smartquill. http://sites.google.com/site/sensecam/smartquill.

[29]

J. O. Wobbrock, A. D. Wilson, and Y. Li. Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In ACM UIST, 2007.

Digital Library

[30]

Xbox. Kinect. http://www.xbox.com/en-US/kinect.

[31]

S. Zhang, C. Yuan, and Y. Zhang. Handwritten character recognition using orientation quantization based on 3d accelerometer. In Mobiquitous, 2008.

Digital Library

[32]

D. Zhuxin, U. C. Wejinya, Z. Shengli, S. Qing, and W. J. Li. Real-time written-character recognition using mems motion sensors: Calibration and experimental results. In ROBIO, 2009.

Digital Library

Cited By

Zhang YSun WLi M(2024)WiRITE: General and Practical Wi-Fi Based Hand-Writing RecognitionIEEE Transactions on Mobile Computing10.1109/TMC.2023.326598823:4(2943-2957)Online publication date: Apr-2024
https://doi.org/10.1109/TMC.2023.3265988
Yin YXie LJiang ZXiao FCao JLu S(2024)A Systematic Review of Human Activity Recognition Based on Mobile Devices: Overview, Progress and TrendsIEEE Communications Surveys & Tutorials10.1109/COMST.2024.335759126:2(890-929)Online publication date: Oct-2025
https://doi.org/10.1109/COMST.2024.3357591
Singh SChaturvedi A(2024)An efficient multi-modal sensors feature fusion approach for handwritten characters recognition using Shapley values and deep autoencoderEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.109225138(109225)Online publication date: Dec-2024
https://doi.org/10.1016/j.engappai.2024.109225
Show More Cited By

Index Terms

Using mobile phones to write in air
1. Human-centered computing
  1. Human computer interaction (HCI)
2. Software and its engineering
  1. Software creation and management
    1. Designing software
      1. Software implementation planning
        Software design techniques
    2. Software development process management

Recommendations

PhonePoint pen: using mobile phones to write in air
MobiHeld '09: Proceedings of the 1st ACM workshop on Networking, systems, and applications for mobile handhelds

The ability to note down small pieces of information, quickly and ubiquitously, can be useful. This paper proposes a system called PhonePoint Pen that uses the in-built accelerometer in mobile phones to recognize human writing. By holding the phone like ...
Integrating isometric joysticks into mobile phones for text entry
CHI EA '06: CHI '06 Extended Abstracts on Human Factors in Computing Systems

We are investigating a new gestural text entry method for mobile phones that uses an isometric joystick and therefore consumes very little physical space. We have created a high-fidelity mobile phone prototype with two embedded isometric joysticks, one ...
Revisiting phone call UIs for multipurpose mobile phones
MobileHCI '13: Proceedings of the 15th international conference on Human-computer interaction with mobile devices and services

While mobile phones made a significant evolution in recent years from single-purpose communication devices to multi-purpose devices, the fundamental design of phone call applications did not evolve accordingly. While its implementation leveraged from ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MobiSys '11: Proceedings of the 9th international conference on Mobile systems, applications, and services

June 2011

430 pages

ISBN:9781450306430

DOI:10.1145/1999995

General Chair:
Ashok Agrawala
University of Maryland, USA
,
Program Chairs:
Mark D. Corner
UMass Amherst, USA
,
David Wetherall
University of Wash. & Intel Labs, USA

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOBILE: ACM Special Interest Group on Mobility of Systems, Users, Data and Computing

In-Cooperation

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 June 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MobiSys'11

Sponsor:

SIGMOBILE

MobiSys'11: The 9th International Conference on Mobile Systems, Applications, and Services

June 28 - July 1, 2011

Maryland, Bethesda, USA

Acceptance Rates

Overall Acceptance Rate 274 of 1,679 submissions, 16%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

122
Total Citations
View Citations
1,407
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)1

Reflects downloads up to 14 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang YSun WLi M(2024)WiRITE: General and Practical Wi-Fi Based Hand-Writing RecognitionIEEE Transactions on Mobile Computing10.1109/TMC.2023.326598823:4(2943-2957)Online publication date: Apr-2024
https://doi.org/10.1109/TMC.2023.3265988
Yin YXie LJiang ZXiao FCao JLu S(2024)A Systematic Review of Human Activity Recognition Based on Mobile Devices: Overview, Progress and TrendsIEEE Communications Surveys & Tutorials10.1109/COMST.2024.335759126:2(890-929)Online publication date: Oct-2025
https://doi.org/10.1109/COMST.2024.3357591
Singh SChaturvedi A(2024)An efficient multi-modal sensors feature fusion approach for handwritten characters recognition using Shapley values and deep autoencoderEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.109225138(109225)Online publication date: Dec-2024
https://doi.org/10.1016/j.engappai.2024.109225
Wu DKe YHe JLi YChen M(2024)Mobile Cloud GamingEncyclopedia of Computer Graphics and Games10.1007/978-3-031-23161-2_77(1163-1169)Online publication date: 5-Jan-2024
https://doi.org/10.1007/978-3-031-23161-2_77
Gao YZeng SZhao JLiu WDong W(2023)AirText: One-Handed Text Entry in the Air for COTS SmartwatchesIEEE Transactions on Mobile Computing10.1109/TMC.2021.313003622:5(2506-2519)Online publication date: 1-May-2023
https://doi.org/10.1109/TMC.2021.3130036
Requena FBarbot NKaddour DPerret E(2023)Orientation Sensing With a Loop Resonator Based on Its Re-Radiation PatternIEEE Sensors Journal10.1109/JSEN.2022.323134223:3(3159-3172)Online publication date: 1-Feb-2023
https://doi.org/10.1109/JSEN.2022.3231342
Regani SWang BHu YLiu K(2023)GWrite: Enabling Through-the-Wall Gesture Writing Recognition Using WiFiIEEE Internet of Things Journal10.1109/JIOT.2022.322431310:7(5977-5991)Online publication date: 1-Apr-2023
https://doi.org/10.1109/JIOT.2022.3224313
Tan XTong JMatsumaru TDutta VHe X(2023)An End-to-End Air Writing Recognition Method Based on TransformerIEEE Access10.1109/ACCESS.2023.332180711(109885-109898)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3321807
Babic TReiterer HHaller M(2022)Understanding and Creating Spatial Interactions with Distant Displays Enabled by Unmodified Off-The-Shelf SmartphonesMultimodal Technologies and Interaction10.3390/mti61000946:10(94)Online publication date: 19-Oct-2022
https://doi.org/10.3390/mti6100094
Chen YHwang WTai TCheng P(2022)Sensor-Based Hand Gesture Detection and Recognition by Key IntervalsApplied Sciences10.3390/app1215741012:15(7410)Online publication date: 23-Jul-2022
https://doi.org/10.3390/app12157410
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents