Keywords

1 Introduction

The development of Web 2.0 technologies has made the WWW a rich source of information to be exploited at business, at all levels of education, and in various communication situations as in the cases of e-Health and e-Government. In parallel, the WWW has become a place where people constantly interact with each other, by posting information (e.g. on blogs or discussion forums), by modifying and enhancing other people’s contributions (e.g. in Wikipedia), and by sharing information (e.g. on Facebook and other social networking websites).

However, as effective as these technologies might be to non-native signers, they seem to create considerable difficulties to native signers, since they require a good command and extensive use of the written language. On the other hand, although the cloud receives vast amounts of video uploads daily, SL videos have two major problems: first, they are not anonymous, and second, they cannot be easily edited and reused in the way written texts can.

To address this problem, a number of research efforts [1, 2] have been reported towards making Web 2.0 accessible for SL users by allowing interactions in SL via the incorporation of real time SL technologies. These technologies are based on available monolingual and bilingual resources with emphasis on dynamic production of SL. The signed utterance is presented via a signing avatar (Fig. 1).

Fig. 1.
figure 1

Sign building and representation environment in the sign-Wiki of the Dicta-Sign demonstrator

Full size image

In this paper, we attempt to demonstrate that Web content may become more accessible to deaf and hard-of-hearing users by combining a set of language technology tools that have been developed for handling SL and vocal/written language. Such tools may incorporate bilingual and monolingual dictionary look-up and fingerspelling facilities as well as a dynamic SL synthesis environment. The proposed architecture exploits bilingual vocal-SL lexicon resources, monolingual SL resources, and language technology (LT) tools including a tagger and a lemmatizer to handle the written form of the vocal language. In the rest of the paper, we present the different components of a workbench with integrated LT tools and technologies. The result was enabling the use by deaf and HoH users of (a) the official educational content platform of the Greek Ministry of Education for the primary and secondary education levels, and (b) the platform’s Graphical User Interface (GUI) characteristics and evaluation procedures. The ultimate aim of this venture was to showcase the accessibility potential of the proposed approach.

2 Integrated Language Technology Tools and Resources

In [3] we extensively discussed the limitations posed by the use of SL videos, while there was also reference to the issues relating to creation of SL resources that are needed to drive SL synthesis engines, along with issues of signing avatar technology and its acceptance by end users. Emphasis was also put on the discussion of various currently implemented interfaces to serve deaf users and suggestions were made as to the tools that would enable better reach to Web content by native signers [3].

The limitations in composing, editing and reusing SL utterances as well as their consequences for Deaf education and communication have been systematically mentioned in the SL literature since the second half of the 20th century. Researchers such as Stokoe [4] and, more recently, the HamNoSyS team [5, 6] and Neidle [7] have proposed different systems for sign transcriptions in an attempt to provide a writing system for SLs in line with the systems available for vocal languages [8]. However, the three-dimensional properties of SLs have prevented wide acceptance of such systems for incorporation by Deaf individuals in everyday practice.

An intuitive way to overcome the set Web 2.0 barriers is to exploit (S)LTs in order to support interfaces which enable signers to easily gain knowledge from electronic text and communicate in SL in a user-friendlier manner.

In this context, many studies have been devoted to improving signing avatar performance in respect to naturalness of signing [9, 10] primarily aiming at higher acceptance rates by Deaf communities. To this end, research has focused on incorporation of principle SL articulation features in avatar signing [11,12,13] exploiting input from SL theoretical linguistic analysis.

The set of tools and resources which enable content accessibility as well as student-student and student-teacher interaction via SL in the platform here presented as the use case to be discussed involve the following specifications and characteristics:

From the part of SL technologies, the tools integrated to the platform entail:

  1. 1

    A bilingual dictionary for the language pair Modern Greek (MG) – Greek Sign Language (GSL), linked with the textbooks uploaded in the platform. Word search is possible by:

    1. 1.1

      double clicking on the encountered words while reading (Fig. 2);

      Fig. 2.
      figure 2

      Association of “unknown” words in a Web text with their equivalent signs by exploitation of the proposed LT toolkit

      Full size image
    2. 1.2

      typing the search item in a search box;

    3. 1.3

      providing search input by means of a virtual fingerspelling keyboard;

    4. 1.4

      copying words and pasting them in the search box. The system then retrieves the correct dictionary entry, irrespective of the inflectional form in the Greek text where originally encountered.

  2. 2

    A dynamic sign phrase synthesis tool [3], which allows composition of signed content on the fly, according to users’ communication needs and makes use of a signing avatar to represent the instantly created content.

  3. 3

    A virtual fingerspelling keyboard of the Greek alphabet characters and the digits 0–9 that enables search and representation of proper names and various number formations via the set of handshapes corresponding to alphabet characters and digits, thus, facilitating accessibility and learning of primarily named entities in all subject areas in the curriculum – from History and Geography to Biology and Mathematics.

  4. 4

    To further facilitate the presentation and reuse of SL content, a link to the multilingual SL resources of the Dicta-Sign FP7 (http://www.sign-lang.uni-hamburg.de/dicta-sign/portal/) project is also available.

From the part of vocal/written LTs, the platform currently makes use of:

  1. 1.

    A segmentation tool that simply segments strings of characters in a text,

  2. 2.

    A tagger, which runs on segmented items and provides labels with grammar information such as morphological tags and tags for syntactic category, and

  3. 3.

    A lemmatizer that associates each string in text with a lemma, which can be checked for match in the bilingual lexicon database.

The abovementioned tools for handling the lexical items found in Greek written texts are all part of the ILSP language tool suite [14], which has been developed by the Institute and is subject to constant improvements. These tools are necessary for the implementation of the integrated accessibility tools, since MG is a highly inflected language and there are many instances of complete change of the form of a lemma in its different syntactic use environments. This, in general, poses a significant extra load to text comprehension by deaf users, who need to become bilingual in the written form of a highly inflected vocal language. Morphological complexity also makes checking different morphological forms against a lexicon a difficult issue demanding numerous filters and raises the retrieval error risk, unless handled in a systematic way by means of LT tools. In the user interface, the initialization of the integrated services is done through the use of help buttons of appropriate shape and size, while color code conventions and pop-up windows for informative or interaction purposes have been employed to ensure that the services are friendly to deaf and HoH users. Moreover, video tooltips are available in GSL in the form of help menus at all stages of use within the Deaf accessibility platform mode (Fig. 3). The integrated deaf accessibility services are initialized by the user and are provided as Add-Ons while browsing through the “Photodentro” and “Digital Educational Content” educational platforms.

Fig. 3.
figure 3

Platform main page with activated deaf accessibility facilities and instructions in GSL

Full size image

The integrated tools are run through a web browser with the help of java applets, currently supporting Mozilla Firefox, Chrome and Internet Explorer. The supported operating systems are Microsoft Windows (XP or above) as well as Mac OS X with Safari browser.

3 Workbench with Integrated LT Tools and Technologies

Within the framework of the implementation presented here, the functionalities facilitating sign search are connected to a look-up environment that can be utilized either in direct link with the viewed text or as an independent dictionary look-up tool activated by demand. The bilingual dictionary, the virtual fingerspelling keyboard and the dynamic sign phrase synthesis tool, described next in more detail, are incorporated in the use case workbench in an interoperable manner in order to support accessibility to text content and communication between platform users [15].

3.1 Bilingual Dictionary: MG-GSL

While browsing through any digital educational text uploaded on the supported educational websites, deaf or HoH users may seek explanation in GSL for any word present in the text. The selected word is then checked against the system’s lexicon of signs using any of the methods described in Sect. 2 above to enter a specific query. The unknown word is looked up in the database of correspondences between written lemmas of MG and the respective GSL sign and, if found, users are presented with information regarding the video lemma representation in GSL, examples of use, synonyms (Fig. 4) linked to senses, or expressions linked to lemmas.

Fig. 4.
figure 4

Presentation of the lemma “barrel”, associated with two forms (synonyms) in GSL

Full size image

The task to be executed, however, is by no means trivial, since in many cases the morphological form of a word in a text differs considerably from the form associated with a headword in a common dictionary, especially in languages with very rich morphology such as MG. Thus, the successful execution of a given query demands an initial processing step of morphological decomposition of the selected word prior to its accurate association with the corresponding entry in the bilingual dictionary, as mentioned in Sect. 2. The search procedure, therefore, entails morphological queries for any given word inserted while it also takes into account any grammatical/semantic differentiation among lemmas in order to filter the search results as, for instance, in the case of the stress position in word sets «θόλος» [th’olos] - «θολός» [thol’os] or semantic differences between two morphologically different types of the same word (e.g. «αφαιρώ» [afer’o]- «αφηρημένος» [afirim’enos]).

The dictionary that supports the accessibility of the textual material currently comprises approximately 10,000 bilingual entries (MG-GSL). A detailed report on its compilation and selection of example sentences along with the methodology followed for the lexicon database creation is provided in [16].

When the use of the dictionary is selected as a standalone platform tool, users may also employ the alphabetically ordered search option. This search option is, in general, considered appropriate for video databases of signs and it has been already applied in some educational and/or e-government internet-based applications [3]. This approach provides the alphabetical ordering of the entailed concepts’ written form, allowing the user to choose the appropriate lemma from the available ordered lists (Fig. 5). This method prevents the delay in the system response caused by searching within the entire content of an extended video database [17].

Fig. 5.
figure 5

Alphabetically ordered lemma look-up

Full size image

In all cases, the information provided incorporates different senses and/or GSL synonym signs with which a MG lexical equivalent may be associated, GSL examples of use for each sense and/or sign available as well as expressions linked to different lemmas.

3.2 Virtual Fingerspelling Keyboard

The fingerspelling keyboard facility comprises a set of virtual keys that correspond to the GSL fingerspelling alphabet. Each key depicts the handshape that represents each letter of the alphabet, while the digits 0–9 are also included. Thus, users can select a sequence of preferred handshapes corresponding to the desired alphanumeric string, while on the screen they can visually inspect the selected sequence being fingerspelled in GSL.

The tool can either run as an external service (Fig. 6) or be interconnected with the lexicon and the dynamic sign synthesis tool as a string input mode of search data (i.e. lemmas). Such tools allow the fingerspelling of proper names [18] and can generally support deaf users while inserting data of the type names, numbers etc. in web forms and also in various other communication situations [19]. Furthermore, when incorporated in the synthetic signing environment, this tool enables the visualization of proper names in the context of a signed utterance and also fingerspelling of “unknown words”, so preventing performance failure. The user interface provides help in GSL in the form of a video tooltip.

Fig. 6.
figure 6

Characters and digits fingerspelling keyboard activated as a search facility both in the dictionary and the synthetic signing environment

Full size image

3.3 Dynamic Sign Synthesis Tool

Sign phrase representation is being performed via a virtual signer through the use of a java applet, which runs in the web browser. The user selects the components of the phrase to be synthesized among the available lexical items that are appropriately coded for synthesis (namely, those containing information not visible to users yet important for synthetic representation). The user interface is designed to allow for different word orderings of the phrase to be synthesized, while the signing phrase may, at this point, consist of up to four components. The composition of new synthetic sign phrases is achieved by selecting the desired phrase components from a list of available, appropriately annotated lexical items [6, 7]. The HamNoSys notation system has been used for the phonological annotation of sign lemmas, along with features for the non-manual activity present in sign formation, while the UEA avatar engine [20] has been used to perform signing. End users interact with the system via a simple search-and-match interface to compose their desired phrases. Phrase components are marked by different color frames indicating which items in the phrase are signed and which are fingerspelled. Users select the desired element by clicking on it and the respective GSL gloss is then included in the sign stream/phrase to be performed by the avatar (Fig. 7). In case a word is not present in the synthesis lexicon, users are provided with the option to fingerspell it. This option proves especially helpful in the case of incorporating proper names in the signed phrase. Furthermore, search results may provide options among which to choose as in the case of possible GSL synonyms.

Fig. 7.
figure 7

Synthetic phrase building and presentation by a signing avatar

Full size image

A drag-and-drop facility, first demonstrated in the Dicta-Sign sign-Wiki [2], allows multiple orderings of phrase components so as to create grammatical structures in GSL. Verifying user choices is important at any stage of this process, so that users can be certain about the content they are creating. When the structuring of the newly built signed phrase is completed (Fig. 7), this phrase is performed by a signing avatar for final verification. Users may select to save, modify, or delete each phrase they have built. They may keep the saved phrases or parts of them for further use depending on their communication needs (e.g. a non-native signer could make use of this utility for language learning purposes).

Since the mainstream school environment is not usually familiar with SL, the sign synthesis tool may also be used by SL illiterate individuals. Thus, in the current implementation, a template-based GSL grammar guide is incorporated in the sign synthesis environment to help non-signers compose grammatically correct GSL utterances. This guide contains a list of core GSL grammatical structures relating to sentence formation, negation, question, and noun-phrase formation, as well as the use of GSL-specific particles (such as the “pa” particle, used to indicate completed actions).

4 Case Study Evaluation

The platform described in this paper underwent different types of evaluation in three stages:

The first stage included a thorough internal technical evaluation process relating to the success of the integration of tools and their effective use in the platform itself. This was carried out by experts on each of the areas involved, who had to evaluate not only each tool individually but also the platform as an integrated whole. Technical evaluation run in parallel to integration work and, in its final stage, focused on issues of robustness and performance stability of the virtual machine (VM) accommodating the platform in the cloud, while receiving huge numbers of parallel requests.

The second evaluation stage involved a small team of five GSL experts, including three native GSL signers, who tried out the platform as end users, acting as informants to provide valuable feedback on the usability and content of the platform. All team members kept a logbook of the part of the content they checked with notes on vocabulary content, GSL example phrases and avatar performance of the signs. The team of experts met regularly to discuss their findings and propose corrections and improvements which were then incorporated in the platform’s database. The same team was consulted regarding the design of the platform GUI.

Finally, the last stage of the evaluation involved real end users themselves. This was carried out during a series of visits at the Deaf School of Argiroupoli, one of the largest Deaf schools established in the area of Athens, including both primary and secondary educational settings. The evaluation was carried out in the form of pilot uses by both teachers and pupils. After becoming familiar with the platform and having used it for an amount of time which allowed them to navigate it with ease, users were asked to provide comments based on their experience with the platform. The comments were collected in the form of subjective evaluation questionnaires with questions on the GUI structure, the provided content and users’ opinions on synthetic signing. The comments that were collected were then studied, grouped (e.g. comments about content or usability, comments with teaching and/or learning implications) and analyzed by the developing team as a means to make possible improvements or additions/extensions.

In general, the platform was positively scored by both user groups. Young pupils tended to find it especially amusing to be able to synthesize signed phrases, while they were pleased with the avatar performance. This result indicates a radical change in signers’ attitude, since previous experience had shown that users are rather concerned as regards SL performance by avatars. The subjective evaluation results presented high scores for the option to compose one’s own wished utterances, which may be one of the reasons that can explain the positive attitude towards avatar performance.

More particularly, search of unknown written content with direct linking to GSL lexicon presentation was among the features which received particularly high scores in respect to their usability as an educational support mechanism.

The most striking result from the part of the teachers was that they considered the platform as a very helpful mechanism to be exploited in bilingual deaf education. To this end, they pointed out that they would like to also see information on grammatical gender of the Greek lemmas in parallel of viewing the Greek equivalent to the presented GSL sign in the vocabulary display window. Finally, the fingerspelling facility was accepted by all users as something natural in the school environment, while several users noted that there are too many proper names to be memorized.

5 Conclusion

Based on the pilot use of the platform and the analysis of user evaluation results, several additions or improvements can be foreseen. For instance, one of the options considered for inclusion in future extensions of the platform is that of search by handshape, so that users will not have to be restricted by using words while searching. This will be utilized following the appropriate grouping of these handshapes within the alphabetical index of the lexical content, so as to limit the number of the search items that are returned by the system.

The composition of synthetic signing phrases may facilitate communication over the Internet and it can also be crucial for class and group work, since it allows for the direct participation and dynamic linguistic message exchange, in a manner similar to what hearing individuals do when writing.

The emerging technology of sign synthesis opens new perspectives with respect to the participation of deaf and HoH individuals in Internet-based everyday activities, including access and retrieval of information and anonymous communication.

Moreover, if advanced SL technology tools and SL resources are combined with standard LT tools and resources for vocal/written language, they may provide workbench environments to radically change the deaf accessibility landscape in the next period. For example, one may envisage the incorporation of sign synthesis environments in machine translation (MT) applications targeting SLs and, thus, opening yet new perspectives as regards their potential and usability [21].