research-article

Public Access

Design and Evaluation of Hybrid Search for American Sign Language to English Dictionaries: Making the Most of Imperfect Sign Recognition

Authors:

Akhter Al Amin,

Matt HuenerfauthAuthors Info & Claims

CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

Article No.: 195, Pages 1 - 13

https://doi.org/10.1145/3491102.3501986

Published: 29 April 2022 Publication History

All formats PDF

Abstract

Searching for the meaning of an unfamiliar sign-language word in a dictionary is difficult for learners, but emerging sign-recognition technology will soon enable users to search by submitting a video of themselves performing the word they recall. However, sign-recognition technology is imperfect, and users may need to search through a long list of possible results when seeking a desired result. To speed this search, we present a hybrid-search approach, in which users begin with a video-based query and then filter the search results by linguistic properties, e.g., handshape. We interviewed 32 ASL learners about their preferences for the content and appearance of the search-results page and filtering criteria. A between-subjects experiment with 20 ASL learners revealed that our hybrid search system outperformed a video-based search system along multiple satisfaction and performance metrics. Our findings provide guidance for designers of video-based sign-language dictionary search systems, with implications for other search scenarios.

Supplementary Material

Supplemental Materials (3491102.3501986-supplemental-materials.zip)

Download
14.72 MB

MP4 File (3491102.3501986-video-figure.mp4)

Video Figure

Download
15.18 MB

References

[1]

Alikhan Abutalipov, Aigerim Janaliyeva, Medet Mukushev, Antonio Cerone, and Anara Sandygulova. 2021. Handshape Classification in a Reverse Dictionary of Sign Languages for the Deaf. In From Data to Models and Back, Juliana Bowles, Giovanna Broccia, and Mirco Nanni (Eds.). Springer International Publishing, Cham, 217–226.

[2]

Azzah Al-Maskari, Mark Sanderson, and Paul Clough. 2007. The Relationship between IR Effectiveness Measures and User Satisfaction. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Amsterdam, The Netherlands) (SIGIR ’07). Association for Computing Machinery, New York, NY, USA, 773–774. https://doi.org/10.1145/1277741.1277902

Digital Library

[3]

Dan Albertson. 2010. Analyzing user interaction with the ViewFinder video retrieval system. Journal of the American Society for Information Science and Technology 61, 2 (2010), 238–252. https://doi.org/10.1002/asi.21257 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/asi.21257

[4]

Oliver Alonzo, Abraham Glasser, and Matt Huenerfauth. 2019. Effect of Automatic Sign Recognition Performance on the Usability of Video-Based Search Interfaces for Sign Language Dictionaries. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility (Pittsburgh, PA, USA) (ASSETS ’19). Association for Computing Machinery, New York, NY, USA, 56–67. https://doi.org/10.1145/3308561.3353791

Digital Library

[5]

Robin Aly, Roeland J.F. Ordelman, Maria Eskevich, Gareth J.F. Jones, and Shu Chen. 2013. Linking inside a Video Collection: What and How to Measure?. In Proceedings of the 22nd International Conference on World Wide Web (Rio de Janeiro, Brazil) (WWW ’13 Companion). Association for Computing Machinery, New York, NY, USA, 457–460. https://doi.org/10.1145/2487788.2487967

Digital Library

[6]

Vasiliki Arpatzoglou, Artemis Kardara, Alexandra Diehl, Barbara Flueckiger, Sven Helmer, and Renato Pajarola. 2021. DanceMoves: A Visual Analytics Tool for Dance Movement Analysis. In EuroVis 2021 - Short Papers, Marco Agus, Christoph Garth, and Andreas Kerren (Eds.). The Eurographics Association, Norrköping, Sweden, 5 pages. https://doi.org/10.2312/evs.20211057

[7]

Vassilis Athitsos, Carol Neidle, Stan Sclaroff, Joan Nash, Alexandra Stefan, Ashwin Thangali, Haijing Wang, and Quan Yuan. 2010. Large lexicon project: American sign language video corpus and sign language indexing/retrieval algorithms. In Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies (CSLT), Vol. 2. European Language Resources Association (ELRA), Valletta, Malta, 11–14.

[8]

Doug Beeferman. 2021. OneLook Dictionary Search. https://www.onelook.com/

[9]

Danielle Bragg, Oscar Koller, Mary Bellard, Larwan Berke, Patrick Boudreault, Annelies Braffort, Naomi Caselli, Matt Huenerfauth, Hernisa Kacorri, Tessa Verhoef, Christian Vogler, and Meredith Ringel Morris. 2019. Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility (Pittsburgh, PA, USA) (ASSETS ’19). Association for Computing Machinery, New York, NY, USA, 16–31. https://doi.org/10.1145/3308561.3353774

Digital Library

[10]

Danielle Bragg, Kyle Rector, and Richard E. Ladner. 2015. A User-Powered American Sign Language Dictionary. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work and Social Computing (Vancouver, BC, Canada) (CSCW ’15). Association for Computing Machinery, New York, NY, USA, 1837–1848. https://doi.org/10.1145/2675133.2675226

Digital Library

[11]

Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101.

[12]

Fabian Bross. 2015. Chereme. In: Hall, T. A. Pompino-Marschall, B. (ed.): Dictionaries of Linguistics and Communication Science (Wörterbücher zur Sprach- und Kommunikationswissenschaft, WSK). Volume: Phonetics and Phonology. Berlin, New York: Mouton de Gruyter.Proceedings of the XVIII EURALEX International Congress 1, 1 (01 2015), 9.

[13]

Fabio Buttussi, Luca Chittaro, and Marco Coppo. 2007. Using Web3D Technologies for Visualization and Search of Signs in an International Sign Language Dictionary. In Proceedings of the Twelfth International Conference on 3D Web Technology (Perugia, Italy) (Web3D ’07). Association for Computing Machinery, New York, NY, USA, 61–70. https://doi.org/10.1145/1229390.1229401

Digital Library

[14]

Carrie J. Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg S. Corrado, Martin C. Stumpe, and Michael Terry. 2019. Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300234

Digital Library

[15]

Necati Cihan Camgöz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. 2018. Neural Sign Language Translation. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. IEEE Computer Society, New York, New York, US, 7784–7793. https://doi.org/10.1109/CVPR.2018.00812

[16]

Necati Cihan Camgöz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2020. Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. IEEE, New York, New York, US, 10020–10030. https://doi.org/10.1109/CVPR42600.2020.01004

[17]

Alvaro Leandro Cavalcante Carneiro, Lucas de Brito Silva, and Denis Henrique Pinheiro Salvadeo. 2021. Efficient sign language recognition system and dataset creation method based on deep learning and image processing. arxiv:2103.12233 [cs.CV]

[18]

Christopher Conly, Zhong Zhang, and Vassilis Athitsos. 2015. An Integrated RGB-D System for Looking up the Meaning of Signs. In Proceedings of the 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments (Corfu, Greece) (PETRA ’15). Association for Computing Machinery, New York, NY, USA, Article 24, 8 pages. https://doi.org/10.1145/2769493.2769534

Digital Library

[19]

Juliet Corbin and Anselm Strauss. 2021. Basics of Qualitative Research (3rd ed.): Techniques and Procedures for Developing Grounded Theory - SAGE Research Methods. https://methods.sagepub.com/book/basics-of-qualitative-research [Online; accessed 19. Mar. 2021].

[20]

Paolo Cremonesi, Mehdi Elahi, and Franca Garzotto. 2017. User interface patterns in recommendation-empowered content intensive multimedia applications. Multimedia Tools and Applications 76, 4 (2017), 5275–5309.

Digital Library

[21]

Tirthankar Dasgupta, Sambit Shukla, Sandeep Kumar, Synny Diwakar, and Anupam Basu. 2008. A Multilingual Multimedia Indian Sign Language Dictionary Tool. In Proceedings of the 6th Workshop on Asian Language Resources. Asian Federation of Natural Language Processing, Hyderabad, India, 53–57. https://www.aclweb.org/anthology/I08-7008

[22]

Eberhard, David M., Gary F. Simons, and Charles D. Fennig (eds.).2021. Sign language. https://www.ethnologue.com/subgroups/sign-language

[23]

Katerina El Raheb and Yannis Ioannidis. 2013. Dance in the World of Data and Objects. In Information Technologies for Performing Arts, Media Access, and Entertainment, Paolo Nesiand Raffaella Santucci (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 192–204.

[24]

Ralph Elliott, Helen Cooper, John Glauert, Richard Bowden, and François Lefebvre-Albaret. 2011. Search-By-Example in Multilingual Sign Language Databases. In Proceedings of the Second International Workshop on Sign Language Translation and Avatar Technology (SLTAT). SLTAT, Dundee, Scotland, 8 pages. http://personal.ee.surrey.ac.uk/Personal/H.Cooper/research/papers/SBE_SLTAT.pdf

[25]

Kirsten Ellis and Jan Carlo Barca. 2012. Exploring Sensor Gloves for Teaching Children Sign Language. Adv. in Hum.-Comp. Int. 2012, Article 12 (Jan. 2012), 1 pages. https://doi.org/10.1155/2012/210507

[26]

National Center for Education Statistics (NCES). 2018. Digest of Education Statistics Number and percentage distribution of course enrollments in languages other than English at degree-granting postsecondary institutions, by language and enrollment level: Selected years, 2002 through 2016. https://nces.ed.gov/programs/digest/d18/tables/dt18_311.80.asp

[27]

José L. Fuertes, Ángel L. González, Gonzalo Mariscal, and Carlos Ruiz. 2006. Bilingual Sign Language Dictionary. In Computers Helping People with Special Needs, Klaus Miesenberger, Joachim Klaus, Wolfgang L. Zagler, and Arthur I. Karshmer (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 599–606.

[28]

David Goldberg, Dennis Looney, and Natalia Lusin. 2015. Enrollments in Languages Other than English in United States Institutions of Higher Education, Fall 2013.

[29]

Google. 2021. Google Scholar. https://scholar.google.com/intl/en/scholar/help.html

[30]

Hong Guo, Zou Shanchen, Chuying Lai, and Hongxin Zhang. 2020. PhyCoVIS: A visual analytic tool of physical coordination for cheer and dance training. Computer Animation and Virtual Worlds 32 (11 2020). https://doi.org/10.1002/cav.1975

[31]

Michael Gusenbauer and Neal R Haddaway. 2020. Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar., and 26 other resources. Research synthesis methods 11, 2 (2020), 181–217.

[32]

Wyatte C Hall, Leonard L Levin, and Melissa L Anderson. 2017. Language deprivation syndrome: A possible neurodevelopmental disorder with sociocultural origins. Social psychiatry and psychiatric epidemiology 52, 6(2017), 761–776.

[33]

Saad Hassan, Oliver Alonzo, Abraham Glasser, and Matt Huenerfauth. 2020. Effect of ranking and precision of results on users’ satisfaction with search-by-video sign-language dictionaries. In Sign Language Recognition, Translation and Production (SLRTP) Workshop-Extended Abstracts, Vol. 4. Computer Vision – ECCV 2020 Workshops, Virtual, 6 pages.

[34]

Saad Hassan, Oliver Alonzo, Abraham Glasser, and Matt Huenerfauth. 2021. Effect of Sign-Recognition Performance on the Usability of Sign-Language Dictionary Search. ACM Trans. Access. Comput. 14, 4, Article 18 (oct 2021), 33 pages. https://doi.org/10.1145/3470650

Digital Library

[35]

Jose L. Hernandez-Rebollar and Erick Mendez. 2004. Interactive American Sign Language Dictionary. In ACM SIGGRAPH 2004 Sketches (Los Angeles, California) (SIGGRAPH ’04). Association for Computing Machinery, New York, NY, USA, 26. https://doi.org/10.1145/1186223.1186256

Digital Library

[36]

Alex Hindle, Jie Shao, Dan Lin, Jiaheng Lu, and Rui Zhang. 2011. Clustering web video search results based on integration of multiple features. World Wide Web 14, 1 (2011), 53–73.

Digital Library

[37]

Robert J Hoffmeister. 2000. A piece of the puzzle: ASL and reading comprehension in deaf children. Mahwah, N.J. : Lawrence Erlbaum Associates, New Jersey, USA. 143–163 pages.

[38]

L. Hollink, G. P. Nguyen, D. C. Koelma, A. T. Schreiber, and M. Worring. 2004. User Strategies in Video Retrieval: A Case Study. In Image and Video Retrieval, Peter Enser, Yiannis Kompatsiaris, Noel E. O’Connor, Alan F. Smeaton, and Arnold W. M. Smeulders (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 6–14.

[39]

Scott B. Huffman and Michael Hochster. 2007. How Well Does Result Relevance Predict Session Satisfaction?. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Amsterdam, The Netherlands) (SIGIR ’07). Association for Computing Machinery, New York, NY, USA, 567–574. https://doi.org/10.1145/1277741.1277839

Digital Library

[40]

Kabil Jaballah and Mohamed Jemni. 2010. Toward Automatic Sign Language Recognition from Web3D Based Scenes. In Proceedings of the 12th International Conference on Computers Helping People with Special Needs(Vienna, Austria) (ICCHP’10). Springer-Verlag, Berlin, Heidelberg, 205–212.

Digital Library

[41]

Vanita Jain, Achin Jain, Abhinav Chauhan, Srinivasu Soma Kotla, and Ashish Gautam. 2021. American Sign Language recognition using Support Vector Machine and Convolutional Neural Network. International Journal of Information Technology 13 (2021), 1–8. https://doi.org/10.1007/s41870-021-00617-x

[42]

Michelle Jenkins. 2004. Evaluation of methodological search filters—a review. Health Information & Libraries Journal 21, 3 (2004), 148–163.

[43]

Michael D. Jones, Harley Hamilton, and James Petmecky. 2015. Mobile Phone Access to a Sign Language Dictionary. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers and Accessibility (Lisbon, Portugal) (ASSETS ’15). Association for Computing Machinery, New York, NY, USA, 331–332. https://doi.org/10.1145/2700648.2811364

Digital Library

[44]

Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. 2017. The Kinetics Human Action Video Dataset. CoRR abs/1705.06950(2017), 5 pages. arxiv:1705.06950http://arxiv.org/abs/1705.06950

[45]

Ksenia Kolykhalova, Antonio Camurri, Gualtiero Völpe, Marcello Sanguineti, Enrico Puppo, and Radosław Niewiadomski. 2015. A multimodal dataset for the analysis of movement qualities in karate martial art. In 2015 7th International Conference on Intelligent Technologies for Interactive Entertainment (INTETAIN). IEEE, IEEE, Piscataway, NJ, 74–78.

[46]

Krishna Kumar. 2020. Song stuck in your head? Just hum to search. https://blog.google/products/search/hum-to-search/

[47]

Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, and Debi Prosad Dogra. 2018. A position and rotation invariant framework for sign language recognition (SLR) using Kinect. Multimedia Tools and Applications 77, 7 (2018), 8823–8846.

Digital Library

[48]

Patricia G Lange. 2007. Publicly private and privately public: Social networking on YouTube. Journal of computer-mediated communication 13, 1 (2007), 361–380.

Digital Library

[49]

J. Lapiak. 2021. Handspeak. https://www.handspeak.com/

[50]

Carman KM Lee, Kam KH Ng, Chun-Hsien Chen, Henry CW Lau, SY Chung, and Tiffany Tsoi. 2021. American sign language recognition and training method with recurrent neural network. Expert Systems with Applications 167 (2021), 114403.

Digital Library

[51]

Mengyang Liu, Yiqun Liu, Jiaxin Mao, Cheng Luo, Min Zhang, and Shaoping Ma. 2018. ”Satisfaction with Failure” or ”Unsatisfied Success”: Investigating the Relationship between Search Success and User Satisfaction. In Proceedings of the 2018 World Wide Web Conference (Lyon, France) (WWW ’18). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 1533–1542. https://doi.org/10.1145/3178876.3186065

Digital Library

[52]

Kai Lukoff, Ulrik Lyngs, Himanshu Zade, J Vera Liao, James Choi, Kaiyue Fan, Sean A Munson, and Alexis Hiniker. 2021. How the Design of YouTube Influences User Sense of Agency. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (to appear). Association for Computing Machinery, New York, NY, USA, 17 pages.

Digital Library

[53]

Klara Majetić and Petra Bago. 2017. Proposing an Instrument for Evaluation of Online Dictionaries of Sign Languages. INTEGRATING ICTIN SOCIETY 1, 1 (2017), 189.

[54]

MDBG. 2021. English to Chinese dictionary. http://www.mdbg.net/

[55]

Daniel Mitchell. 2021. British Sign Language BSL Dictionary. https://www.signbsl.com/

[56]

Ross Mitchell, Travas Young, Bellamie Bachleda, and Michael Karchmer. 2006. How Many People Use ASL in the United States? Why Estimates Need Updating. Sign Language Studies 6 (03 2006). https://doi.org/10.1353/sls.2006.0019

[57]

J Murray. 2020. World Federation of the deaf.http://wfdeaf.org/our-work/

[58]

Carol Neidle and Christian Vogler. 2012. A new web interface to facilitate access to corpora: Development of the ASLLRP data access interface (DAI). In Proc. 5th Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon, LREC. Citeseer, OpenBU, Istanbul, Turkey, 8 pages. https://open.bu.edu/handle/2144/31886

[59]

Aiden Nibali, Joshua Millward, Zhen He, and Stuart Morgan. 2021. ASPset: An outdoor sports pose video dataset with 3D keypoint annotations. Image and Vision Computing 111 (2021), 104196.

[60]

Asanka G Perera, Yee Wei Law, Titilayo T Ogunwa, and Javaan Chahl. 2020. A multiviewpoint outdoor dataset for human action recognition. IEEE Transactions on Human-Machine Systems 50, 5 (2020), 405–413.

[61]

Junfu Pu, Wengang Zhou, and Houqiang Li. 2018. Dilated Convolutional Network with Iterative Optimization for Continuous Sign Language Recognition. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18. International Joint Conferences on Artificial Intelligence Organization, Stockholm, Sweden, 885–891. https://doi.org/10.24963/ijcai.2018/123

[62]

Fanchao Qi, Lei Zhang, Yanhui Yang, Zhiyuan Liu, and Maosong Sun. 2020. WantWords: An Open-source Online Reverse Dictionary System, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations 57, 175–181. https://doi.org/10.18653/v1/2020.emnlp-demos.23

[63]

Razieh Rastgoo, Kourosh Kiani, and Sergio Escalera. 2021. Sign Language Recognition: A Deep Survey. Expert Systems with Applications 164 (2021), 113794. https://doi.org/10.1016/j.eswa.2020.113794

[64]

Kishore K Reddy and Mubarak Shah. 2013. Recognizing 50 human action categories of web videos. Machine vision and applications 24, 5 (2013), 971–981.

[65]

Tânia Rocha, J Martins, F Branco, and R Gonçalves. 2017. Evaluating youtube platform usability by people with intellectual disabilities (a user experience case study performed in a six-month period). Journal of Information Systems Engineering & Management 2, 1 (2017), 5.

[66]

Jerry Schnepp, Rosalee Wolfe, Gilbert Brionez, Souad Baowidan, Ronan Johnson, and John McDonald. 2020. Human-Centered Design for a Sign Language Learning Application. In Proceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive Environments (Corfu, Greece) (PETRA ’20). Association for Computing Machinery, New York, NY, USA, Article 60, 5 pages. https://doi.org/10.1145/3389189.3398007

Digital Library

[67]

Akari Shimono, Yuki Kakui, and Toshihiko Yamasaki. 2020. Automatic YouTube-Thumbnail Generation and Its Evaluation. In Proceedings of the 2020 Joint Workshop on Multimedia Artworks Analysis and Attractiveness Computing in Multimedia (Dublin, Ireland) (MMArt-ACM ’20). Association for Computing Machinery, New York, NY, USA, 25–30. https://doi.org/10.1145/3379173.3393711

Digital Library

[68]

ShuR. 2021. SLintoDictionary. http://slinto.com/us

[69]

Sotirios Stasinopoulos and Petros Maragos. 2012. Human action recognition using Histographic methods and hidden Markov models for visual martial arts applications. In 2012 19th IEEE International Conference on Image Processing. IEEE, Piscataway, NJ, 745–748. https://doi.org/10.1109/ICIP.2012.6466967

[70]

William C Stokoe. 1980. Sign language structure. Annual Review of Anthropology 9 (1980), 365–390.

[71]

P Kumar Sugandhi and Sanmeet Kaur. 2018. Online multilingual dictionary using Hamburg notation for avatar-based Indian sign language generation system. International Journal of Cognitive and Language Sciences 12, 8(2018), 120–127.

[72]

Piet Swanepoel. 2001. Dictionary quality and dictionary design: a methodology for improving the functional quality of dictionaries. Lexikos 11(2001), 160–190.

[73]

Piet Swanepoel. 2008. Towards a framework for the description and evaluation of dictionary evaluation criteria. Lexikos 18(2008), 207–231.

[74]

Nazif Can Tamer and Murat Saraçlar. 2020. Improving Keyword Search Performance in Sign Language with Hand Shape Features. In Computer Vision – ECCV 2020 Workshops, Adrien Bartoli and Andrea Fusiello (Eds.). Springer International Publishing, Cham, 322–333.

[75]

Damien Tardieu, Xavier Siebert, Barbara Mazzarino, Ricardo Chessini, Julien Dubois, Stéphane Dupont, Giovanna Varni, and Alexandra Visentin. 2010. Browsing a dance video collection: dance analysis and interface design. Journal on Multimodal User Interfaces 4, 1 (2010), 37–46.

[76]

Shuhei Tsuchida, Satoru Fukayama, and Masataka Goto. 2019. Query-by-Dancing: A Dance Music Retrieval System Based on Body-Motion Similarity. In MultiMedia Modeling, Ioannis Kompatsiaris, Benoit Huet, Vasileios Mezaris, Cathal Gurrin, Wen-Huang Cheng, and Stefanos Vrochidis (Eds.). Springer International Publishing, Cham, 251–263.

[77]

Haijing Wang, Alexandra Stefan, Sajjad Moradi, Vassilis Athitsos, Carol Neidle, and Farhad Kamangar. 2012. A System for Large Vocabulary Sign Search. In Trends and Topics in Computer Vision, Kiriakos N. Kutulakos (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 342–353.

[78]

Zhe Wang. 2012. Similarity Search with Multimodal Data. Ph.D. Dissertation. Princeton University. https://ezproxy.rit.edu/login?url=https://www.proquest.com/dissertations-theses/similarity-search-with-multimodal-data/docview/963697731/se-2?accountid=108 Copyright - Database copyright ProQuest LLC; ProQuest does not claim copyright in the individual underlying works; Last updated - 2021-05-26.

[79]

Michael M Ward. 2013. Sense of control and self-reported health in a population-based sample of older Americans: assessment of potential confounding by affect, personality, and social support. International journal of behavioral medicine 20, 1 (2013), 140–147.

[80]

Kimberly A. Weaver and Thad Starner. 2011. We Need to Communicate! Helping Hearing Parents of Deaf Children Learn American Sign Language. In The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility (Dundee, Scotland, UK) (ASSETS ’11). Association for Computing Machinery, New York, NY, USA, 91–98. https://doi.org/10.1145/2049536.2049554

Digital Library

[81]

S. Wilcox, J. Scheibman, D. Wood, D. Cokely, and W. C. Stokoe. 1994. Multimedia Dictionary of American Sign Language. In Proceedings of the First Annual ACM Conference on Assistive Technologies (Marina Del Rey, California, USA) (Assets ’94). Association for Computing Machinery, New York, NY, USA, 9–16. https://doi.org/10.1145/191028.191031

Digital Library

[82]

Shicheng Xu, Huan Li, Xiaojun Chang, Shoou-I Yu, Xingzhong Du, Xuanchong Li, Lu Jiang, Zexi Mao, Zhenzhong Lan, Susanne Burger, and Alexander Hauptmann. 2015. Incremental Multimodal Query Construction for Video Search. In Proceedings of the 5th ACM on International Conference on Multimedia Retrieval (Shanghai, China) (ICMR ’15). Association for Computing Machinery, New York, NY, USA, 675–678. https://doi.org/10.1145/2671188.2749413

Digital Library

[83]

Polina Yanovich, Carol Neidle, and Dimitris Metaxas. 2016. Detection of Major ASL Sign Types in Continuous Signing For ASL Recognition. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16). European Language Resources Association (ELRA), Portorož, Slovenia, 3067–3073. https://www.aclweb.org/anthology/L16-1490

[84]

Yuya Yoshikawa, Jiaqing Lin, and Akikazu Takeuchi. 2018. STAIR Actions: A Video Dataset of Everyday Home Actions. CoRR abs/1804.04326(2018), 17 pages. arxiv:1804.04326http://arxiv.org/abs/1804.04326

[85]

YouTube. 2021. Advanced search (Filters) - Computer - YouTube Help. https://support.google.com/youtube/answer/111997

[86]

Jin Yuan, Yi-Liang Zhao, Huanbo Luan, Meng Wang, and Tat-Seng Chua. 2014. Memory Recall Based Video Search: Finding Videos You Have Seen before Based on Your Memory. ACM Trans. Multimedia Comput. Commun. Appl. 10, 2, Article 21 (Feb. 2014), 21 pages. https://doi.org/10.1145/2534409

Digital Library

[87]

NA Yurko, IM Styfanyshyn, UM Protsenko, and OV Romanchuk. 2020. English dictionary apps: the features and benefits.

[88]

Zahoor Zafrulla, Helene Brashear, Thad Starner, Harley Hamilton, and Peter Presti. 2011. American Sign Language Recognition with the Kinect. In Proceedings of the 13th International Conference on Multimodal Interfaces (Alicante, Spain) (ICMI ’11). Association for Computing Machinery, New York, NY, USA, 279–286. https://doi.org/10.1145/2070481.2070532

Digital Library

[89]

Weichen Zhang, Zhiguang Liu, Liuyang Zhou, Howard Leung, and Antoni B Chan. 2017. Martial arts, dancing and sports dataset: A challenging stereo and multi-view dataset for 3d human pose estimation. Image and Vision Computing 61 (2017), 22–39.

Digital Library

[90]

Weichen Zhang, Zhiguang Liu, Liuyang Zhou, Howard Leung, and Antoni B. Chan. 2017. Martial Arts, Dancing and Sports dataset: A challenging stereo and multi-view dataset for 3D human pose estimation. Image and Vision Computing 61 (2017), 22–39. https://doi.org/10.1016/j.imavis.2017.02.002

Digital Library

Cited By

Hassan Sde Lacerda Pataca CAmin ANourian LNavarro DLee SGordon AWatkins MTigwell GHuenerfauth M(2024)Exploring the Benefits and Applications of Video-Span Selection and Search for Real-Time Support in Sign Language Video Comprehension among ASL LearnersACM Transactions on Accessible Computing10.1145/369064717:3(1-35)Online publication date: 4-Oct-2024
https://dl.acm.org/doi/10.1145/3690647
Hassan SDe Lacerda Pataca CNourian LTigwell GDavis BSilver Wagman W(2024)Designing and Evaluating an Advanced Dance Video Comprehension Tool with In-situ Move Identification CapabilitiesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642710(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642710
Hossain EBao ANewman KMann MWang HLi YKurumada CHall WBai Z(2023)Supporting ASL Communication Between Hearing Parents and Deaf ChildrenProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3614511(1-5)Online publication date: 22-Oct-2023
https://dl.acm.org/doi/10.1145/3597638.3614511
Show More Cited By

Index Terms

Design and Evaluation of Hybrid Search for American Sign Language to English Dictionaries: Making the Most of Imperfect Sign Recognition
1. Human-centered computing
  1. Accessibility
    1. Accessibility systems and tools
2. Information systems
  1. Information retrieval
    1. Users and interactive retrieval
      1. Search interfaces

Recommendations

Support in the Moment: Benefits and use of video-span selection and search for sign-language video comprehension among ASL learners
ASSETS '22: Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility

As they develop comprehension skills, American Sign Language (ASL) learners often view challenging ASL videos, which may contain unfamiliar signs. Current dictionary tools require students to isolate a single sign they do not understand and input a ...
Effect of Sign-recognition Performance on the Usability of Sign-language Dictionary Search
Advances in sign-language recognition technology have enabled researchers to investigate various methods that can assist users in searching for an unfamiliar sign in ASL using sign-recognition technology. Users can generate a query by submitting a video ...
Sign Spotter: Design and Initial Evaluation of an Automatic Video-Based American Sign Language Dictionary System
ASSETS '23: Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility

Searching unfamiliar American Sign Language (ASL) words in a dictionary is challenging for learners, as it involves recalling signs from memory and providing specific linguistic details. Fortunately, the emergence of sign-recognition technology will soon ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

April 2022

10459 pages

ISBN:9781450391573

DOI:10.1145/3491102

Editors:
Simone Barbosa
PUC-Rio, Brazil
,
Cliff Lampe
University of Michigan, USA
,
Caroline Appert
Université Paris-Saclay, France
,
David A. Shamma
Toyota Research Institute, USA
,
Steven Drucker
Microsoft Research, USA
,
Julie Williamson
University of Glasgow, UK
,
Koji Yatani
University of Tokyo, Japan

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Science Foundation

Conference

CHI '22

Sponsor:

SIGCHI

CHI '22: CHI Conference on Human Factors in Computing Systems

April 29 - May 5, 2022

LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
616
Total Downloads

Downloads (Last 12 months)253
Downloads (Last 6 weeks)23

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hassan Sde Lacerda Pataca CAmin ANourian LNavarro DLee SGordon AWatkins MTigwell GHuenerfauth M(2024)Exploring the Benefits and Applications of Video-Span Selection and Search for Real-Time Support in Sign Language Video Comprehension among ASL LearnersACM Transactions on Accessible Computing10.1145/369064717:3(1-35)Online publication date: 4-Oct-2024
https://dl.acm.org/doi/10.1145/3690647
Hassan SDe Lacerda Pataca CNourian LTigwell GDavis BSilver Wagman W(2024)Designing and Evaluating an Advanced Dance Video Comprehension Tool with In-situ Move Identification CapabilitiesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642710(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642710
Hossain EBao ANewman KMann MWang HLi YKurumada CHall WBai Z(2023)Supporting ASL Communication Between Hearing Parents and Deaf ChildrenProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3614511(1-5)Online publication date: 22-Oct-2023
https://dl.acm.org/doi/10.1145/3597638.3614511
Bohacek MHassan S(2023)Sign Spotter: Design and Initial Evaluation of an Automatic Video-Based American Sign Language Dictionary SystemProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3614497(1-5)Online publication date: 22-Oct-2023
https://dl.acm.org/doi/10.1145/3597638.3614497
Coster MDambre J(2023)Querying A Sign Language Dictionary with Videos Using Dense Vector Search2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW)10.1109/ICASSPW59220.2023.10193531(1-5)Online publication date: 4-Jun-2023
https://doi.org/10.1109/ICASSPW59220.2023.10193531

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents