research-article

A User-Powered American Sign Language Dictionary

Authors:

Danielle Bragg,

Richard E. LadnerAuthors Info & Claims

CSCW '15: Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing

Pages 1837 - 1848

https://doi.org/10.1145/2675133.2675226

Published: 28 February 2015 Publication History

Abstract

Students learning American Sign Language (ASL) have trouble searching for the meaning of unfamiliar signs. ASL signs can be differentiated by a small set of simple features including hand shape, orientation, location, and movement. In a feature-based ASL-to-English dictionary, users search for a sign by providing a query, which is a set of observed features. Because there is natural variability in the way signs are executed, and observations are error-prone, an approach other than exact matching of features is needed. We propose ASL-Search, an ASL-to-English dictionary entirely powered by its users. ASL-Search utilizes Latent Semantic Analysis (LSA) on a database of feature-based user queries to account for variability. To demonstrate ASL-Search's viability, we created ASL-Flash, a learning tool that presents online flashcards to ASL students and provides query data. Our simulations on this data serve as a proof of concept, demonstrating that our dictionary's performance improves with use and performs well for users with varied levels of ASL experience.

References

[1]

Agichtein, E., Brill, E., and Dumais, S. Improving web search ranking by incorporating user behavior information. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, ACM (2006), 19--26.

Digital Library

[2]

Agrahri, A. K., Manickam, D. A. T., and Riedl, J. Can people collaborate to improve the relevance of search results? In Proceedings of the 2008 ACM conference on Recommender systems, ACM (2008), 283--286.

Digital Library

[3]

Alonso, O., Rose, D. E., and Stewart, B. Crowdsourcing for relevance evaluation. In ACM SIGIR Forum, vol. 42, ACM (2008), 9--15.

Digital Library

[4]

Aslpro, May 2014. http://www.aslpro.com.

[5]

Aslwrite - support american sign language through writing!, May 2013. http://www.aslwrite.com/.

[6]

Bao, S., Xue, G., Wu, X., Yu, Y., Fei, B., and Su, Z. Optimizing web search using social annotations. In Proceedings of the 16th international conference on World Wide Web, ACM (2007), 501--510.

Digital Library

[7]

Cooper, H., Pugeault, N., and Bowden, R. Reading the signs: A video based sign dictionary. In Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, IEEE (2011), 914--919.

[8]

Deerwester, S. C., Dumais, S. T., Landauer, T. K., Furnas, G. W., and Harshman, R. A. Indexing by latent semantic analysis. Journal of the Association for Information Science and Technology (JASIS) 41, 6 (1990), 391--407.

[9]

Elliott, R., Cooper, H., Ong, E.-J., Glauert, J., Bowden, R., and Lefebvre-Albaret, F. Search-by-example in multilingual sign language databases. In Procs. of Second International Workshop on Sign Language Translation and Avatar Technology (SLTAT) (2011).

[10]

Furman, N., Goldberg, D., and Lusin, N. Enrollments in languages other than english in USA institutions of higher education, fall 2009. In Modern Language Association, ERIC (2010).

[11]

Giordano, T. Toward a crowdsourced model for ASL translations of Shakespeare's works. PhD thesis, California State University, 2013.

[12]

Grady, C., and Lease, M. Crowdsourcing document relevance assessment with mechanical turk. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon's mechanical turk, Association for Computational Linguistics (2010), 172--179.

Digital Library

[13]

Imagawa, K., Lu, S., and Igi, S. Color-based hands tracking system for sign language recognition. In Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, IEEE (1998), 462--467.

Digital Library

[14]

Järvelin, K., and Kekäläinen, J. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS) 20, 4 (2002), 422--446.

Digital Library

[15]

Joachims, T. Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, ACM (2002), 133--142.

Digital Library

[16]

Joachims, T., and Radlinski, F. Search engines that learn from implicit feedback. IEEE Computer 40, 8 (2007), 34--40.

Digital Library

[17]

Kamps, J., Geva, S., Peters, C., Sakai, T., Trotman, A., and Voorhees, E. Report on the sigir 2009 workshop on the future of ir evaluation. In ACM SIGIR Forum, vol. 43, ACM (2009), 13--23.

Digital Library

[18]

Kazai, G. In search of quality in crowdsourcing for search engine evaluation. In Advances in information retrieval, vol. 6611 of Lecture Notes in Computer Science. Springer, 2011, 165--176.

Digital Library

[19]

Kazai, G., Kamps, J., Koolen, M., and Milic-Frayling, N. Crowdsourcing for book search evaluation: impact of hit design on comparative system ranking. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, ACM (2011), 205--214.

Digital Library

[20]

Kim, J.-H., Thang, N. D., and Kim, T.-S. 3-d hand motion tracking and gesture recognition using a data glove. In Industrial Electronics, 2009. ISIE 2009. IEEE International Symposium on, IEEE (2009), 1013--1018.

[21]

Kim, J.-S., Jang, W., and Bien, Z. A dynamic gesture recognition system for the korean sign language (ksl). Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on 26, 2 (1996), 354--359.

Digital Library

[22]

Landauer, T. K., Foltz, P. W., and Laham, D. An introduction to latent semantic analysis. Discourse processes 25, 2-3 (1998), 259--284.

[23]

Lasecki, W., Miller, C., Sadilek, A., Abumoussa, A., Borrello, D., Kushalnagar, R., and Bigham, J. Real-time captioning by groups of non-experts. In Proceedings of the 25th annual ACM symposium on User interface software and technology, ACM (2012), 23--34.

Digital Library

[24]

Le, J., Edmonds, A., Hester, V., and Biewald, L. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In SIGIR 2010 workshop on crowdsourcing for search evaluation (2010), 21--26.

[25]

Liang, R.-H., and Ouhyoung, M. A real-time continuous gesture recognition system for sign language. In Automatic Face and Gesture Recognition, 1998. Proceedings. Third IEEE International Conference on, IEEE (1998), 558--567.

Digital Library

[26]

Ong, E.-J., Cooper, H., Pugeault, N., and Bowden, R. Sign language recognition using sequential pattern trees. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, IEEE (2012), 2200--2207.

Digital Library

[27]

Parent, G., and Eskenazi, M. Clustering dictionary definitions using amazon mechanical turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Association for Computational Linguistics (2010), 21--29.

Digital Library

[28]

Post, M., Callison-Burch, C., and Osborne, M. Constructing parallel corpora for six indian languages via crowdsourcing. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Association for Computational Linguistics (2012), 401--409.

Digital Library

[29]

Radlinski, F., and Joachims, T. Query chains: learning to rank from implicit feedback. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, ACM (2005), 239--248.

Digital Library

[30]

Rui, Y., Huang, T. S., and Chang, S.-F. Image retrieval: Current techniques, promising directions, and open issues. Journal of visual communication and image representation 10, 1 (1999), 39--62.

[31]

Signwriting for sign languages, May 2013. http://www.signwriting.org/.

[32]

Starner, T., and Pentland, A. Real-time american sign language recognition from video using hidden markov models. In Motion-Based Recognition. Springer, 1997, 227--243.

[33]

Stokoe, W. C., Casterline, D. C., and Croneberg, C. G. A dictionary of American Sign Language on linguistic principles. Linstok Press Silver Spring, 1976.

[34]

Tennant, R. A., and Brown, M. G. The American sign language handshape dictionary. Gallaudet University Press, 1998.

[35]

Trainor, C. Open source, crowd source: harnessing the power of the people behind our libraries. Program: electronic library and information systems 43, 3 (2009), 288--298.

[36]

The ultimate american sign language dictionary, May 2013. http://idrt.com/store/the-ultimate-american-sign-language-dictionary.

[37]

von Ahn, L. Duolingo: learn a language for free while helping to translate the web. In Proceedings of the 2013 international conference on Intelligent user interfaces (IUI), ACM Press (2013), 1--2.

Digital Library

[38]

Wang, H., Stefan, A., Moradi, S., Athitsos, V., Neidle, C., and Kamangar, F. A system for large vocabulary sign search. In Trends and Topics in Computer Vision. Springer, 2012, 342--353.

Digital Library

[39]

Wang, R. Y., and Popović, J. Real-time hand-tracking with a color glove. In ACM Transactions on Graphics (TOG), vol. 28, ACM (2009), 63.

Digital Library

[40]

Yang, M.-H., and Ahuja, N. Recognizing hand gestures using motion trajectories. In Face Detection and Gesture Recognition for Human-Computer Interaction. Springer, 2001, 53--81.

[41]

Zafrulla, Z., Brashear, H., Starner, T., Hamilton, H., and Presti, P. American sign language recognition with the kinect. In Proceedings of the 13th international conference on multimodal interfaces, ACM (2011), 279--286.

Digital Library

[42]

Zinza, J. E. Master ASL! Sign Media Inc, 2006.

Cited By

Hassan Sde Lacerda Pataca CAmin ANourian LNavarro DLee SGordon AWatkins MTigwell GHuenerfauth M(2024)Exploring the Benefits and Applications of Video-Span Selection and Search for Real-Time Support in Sign Language Video Comprehension among ASL LearnersACM Transactions on Accessible Computing10.1145/369064717:3(1-35)Online publication date: 4-Oct-2024
https://dl.acm.org/doi/10.1145/3690647
Huamani-Malca JBejarano G(2024)Comparing Incremental Learning Approaches for a Growing Sign Language DictionaryInformation Management and Big Data10.1007/978-3-031-63616-5_7(97-106)Online publication date: 29-Jun-2024
https://doi.org/10.1007/978-3-031-63616-5_7
Desai ABerger LMinakov FMilan VSingh CPumphrey KLadner RDaumé HLu ACaselli NBragg DOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)ASL citizenProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3669482(76893-76907)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3669482
Show More Cited By

Index Terms

A User-Powered American Sign Language Dictionary

Recommendations

Design and Evaluation of Hybrid Search for American Sign Language to English Dictionaries: Making the Most of Imperfect Sign Recognition
CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

Searching for the meaning of an unfamiliar sign-language word in a dictionary is difficult for learners, but emerging sign-recognition technology will soon enable users to search by submitting a video of themselves performing the word they recall. ...
Effect of Automatic Sign Recognition Performance on the Usability of Video-Based Search Interfaces for Sign Language Dictionaries
ASSETS '19: Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility

Researchers have investigated various methods to help users search for the meaning of an unfamiliar word in American Sign Language (ASL). Some are based on sign-recognition technology, e.g. a user performs a word into a webcam and obtains a list of ...
Effect of Sign-recognition Performance on the Usability of Sign-language Dictionary Search
Advances in sign-language recognition technology have enabled researchers to investigate various methods that can assist users in searching for an unfamiliar sign in ASL using sign-recognition technology. Users can generate a query by submitting a video ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CSCW '15: Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing

February 2015

1956 pages

ISBN:9781450329224

DOI:10.1145/2675133

General Chairs:
Dan Cosley
Cornell University, USA
,
Andrea Forte
Drexel University, USA
,
Program Chairs:
Luigina Ciolfi
Sheffield Hallam University, UK
,
David McDonald
University of Washington, USA

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 February 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

CSCW '15

Sponsor:

SIGCHI

CSCW '15: Computer Supported Cooperative Work and Social Computing

March 14 - 18, 2015

BC, Vancouver, Canada

Acceptance Rates

CSCW '15 Paper Acceptance Rate 161 of 575 submissions, 28%;

Overall Acceptance Rate 2,235 of 8,521 submissions, 26%

Upcoming Conference

CSCW '25

Sponsor:
sigchi

Computer-Supported Cooperative Work and Social Computing

October 18 - 22, 2025

Bergen , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
589
Total Downloads

Downloads (Last 12 months)58
Downloads (Last 6 weeks)5

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hassan Sde Lacerda Pataca CAmin ANourian LNavarro DLee SGordon AWatkins MTigwell GHuenerfauth M(2024)Exploring the Benefits and Applications of Video-Span Selection and Search for Real-Time Support in Sign Language Video Comprehension among ASL LearnersACM Transactions on Accessible Computing10.1145/369064717:3(1-35)Online publication date: 4-Oct-2024
https://dl.acm.org/doi/10.1145/3690647
Huamani-Malca JBejarano G(2024)Comparing Incremental Learning Approaches for a Growing Sign Language DictionaryInformation Management and Big Data10.1007/978-3-031-63616-5_7(97-106)Online publication date: 29-Jun-2024
https://doi.org/10.1007/978-3-031-63616-5_7
Desai ABerger LMinakov FMilan VSingh CPumphrey KLadner RDaumé HLu ACaselli NBragg DOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)ASL citizenProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3669482(76893-76907)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3669482
Bohacek MHassan S(2023)Sign Spotter: Design and Initial Evaluation of an Automatic Video-Based American Sign Language Dictionary SystemProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3614497(1-5)Online publication date: 22-Oct-2023
https://dl.acm.org/doi/10.1145/3597638.3614497
Bragg DGlasser AMinakov FCaselli NThies W(2022)Exploring Collection of Sign Language Videos through CrowdsourcingProceedings of the ACM on Human-Computer Interaction10.1145/35556276:CSCW2(1-24)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555627
Hassan SAmin Ade Lacerda Pataca CNavarro DGordon ALee SHuenerfauth M(2022)Support in the Moment: Benefits and use of video-span selection and search for sign-language video comprehension among ASL learnersProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3544883(1-14)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1145/3517428.3544883
Glasser AMinakov FBragg D(2022)ASL Wiki: An Exploratory Interface for Crowdsourcing ASL TranslationsProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3544827(1-13)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1145/3517428.3544827
Hassan SElsweiler DKruschwitz ULudwig B(2022)Designing and Experimentally Evaluating a Video-based American Sign Language Look-up SystemProceedings of the 2022 Conference on Human Information Interaction and Retrieval10.1145/3498366.3505804(383-386)Online publication date: 14-Mar-2022
https://dl.acm.org/doi/10.1145/3498366.3505804
Hassan SAmin AGordon ALee SHuenerfauth M(2022)Design and Evaluation of Hybrid Search for American Sign Language to English Dictionaries: Making the Most of Imperfect Sign RecognitionProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501986(1-13)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3501986
Hassan SAlonzo OGlasser AHuenerfauth M(2021)Effect of Sign-recognition Performance on the Usability of Sign-language Dictionary SearchACM Transactions on Accessible Computing10.1145/347065014:4(1-33)Online publication date: 15-Oct-2021
https://dl.acm.org/doi/10.1145/3470650
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents