English (pdf)
Article in xml format
Article references
Automatic translation
Send this article by e-mail
Cited by SciELO 
Cited by Google 
Similars in
SciELO 
Similars in Google 
Permalink
Introduction
At present, advances in technology have been increasing, given the capacity of the human being in terms of knowledge and experience throughout life, have made it possible to generate communication alternatives, by virtue of the need to interact with people who handle different languages and in this case with deaf-mutes, which through the inclusion of different types of tools, mechanisms, services, among others, it has enabled them to promote and contribute instruments to society by focusing their efforts on the common good.
Hearing impairment is represented in approximately 5% of the world’s population, or 446 million people, to be more specific within South America; Similarly, 1% of the population in Colombia, from a sample of 500,000 people present with deafness 1.
From the above, there is a need to reduce the communication gap between deaf-mutes and speakers. For this reason, the proposal arises to create an application that allows to have an alternative of communication translating sign language to text from a mobile device, based on identified reference points of the hand. Likewise, allowing modules where learning material is provided with free access and based on intangible, cultural and linguistic recognition Colombian national.
Related works
To carry out this work it was necessary to make a characterization of different projects that were similar in order to have references about what has been done and what has not been done in order to arrive at the construction of the proposal presented in this article. The following studies are highlighted:
Regarding the article Inclusive strategy mediated by ICT for the improvement of communication of people with hearing and vocal disabilities, presented by the authors 2. This article presents a technological tool, based on a mobile application aimed at people with hearing and speech disabilities; its main objective was to improve the communication process between people with this type of disability and those who do not. It is important to note that this strategy worked with sign language based on the Colombian dactyological alphabet. This work was taken as a reference to strengthen the theoretical and functional base as regards the structural construction of the proposal.
Disruptive technological tool for social inclusion in deaf people, this article prepared by 3, presents a mobile application aimed at deaf people, the basis of the application is to increase the level of communication between deaf people and hearing people; which worked with sign language based on the Colombian dactyological alphabet, giving priority to the language of deaf people; to make a translation into the Spanish language, this is why it includes the use of abbreviated language through moving images and also add predictions of text on the keyboard through the use of files with extension .png. Finally, the respective tests of the application were carried out, from which some suggestions were obtained as improvements, which were already applied and adjusted in the version included in this work.
Indian sign language conversion system using an Android application, presented by 4. It is a system that uses a color model Hue, Saturation, HSV intensity for tracking and segmentation of hands. Thanks to this, it was possible to do a neural network training for data classification. The application captures images of the gestures and in a web hosting are taken as input to perform processing in the neural network. From the study of this project, it was possible to highlight the use of the REST API for handling the information generated from the scanning of the images; this meeting the need to release the possible flow considerable processed data.
Translator of text and voice to Ecuadorian sign language through an avatar implemented for Android devices, according to the authors 5. The translator aims to become a tool for learning Ecuadorian sign language and thus help deaf and dumb people to communicate within this family and its environment. This system has a friendly interface with an avatar of human appearance which performs the signals and the respective movements of 120 frequently used words; this translator has been developed in Android Studio to be run on Android devices. On the main screen of the interface menu you have two input options per keyboard of one, two or three words; or by voice recognition of a phrase containing three words, in both cases the phrase has been limited to 25 characters; is intended then to provide a tool to contribute to the learning of deaf and dumb people both children and adults and mainly of those who without having any disability are interested in learning Ecuadorian sign language LSEC in an entertaining and simple. However, it is important to understand that sign language as a means of communication, the implementation of identification and translation technologies, allow reducing communication gaps, improve people’s quality of life and provide progress in the implementation of new technologies
Android application to promote communication in Cuban sign language, proposed by 6. The android app developed to promote communication in Cuban sign language, containing 122 videos, nourishes families with hearing impaired children at the José Antonio Echevarría Bianchi school with the necessary vocabulary to interact with their children. It also informs them of the characteristics they possess, as well as the rules strategies for better communication. This work was taken as a reference in terms of the structure and organization of the contents in order to present them in a pleasant and attractive way for users.
The prototype of the mobile application for learning Colombian Sign Language on Android by the author 7 provides access to people with and without hearing limitations. It stands out for its didactic approach, using gesture-spatial expression. The application is distinguished by creating 3 modules with multiple learning units, unlike other research. The tool's acceptance was positive. The project is characterized by being complete, educational, and inclusive, prioritizing an intuitive design for the hearing-impaired population. This work contributed the importance of making a design that was practical and easy to use for users.
Design of an electronic glove for the interpretation and translation of sign language in people with hearing disabilities using Arduino technology and a display interface through an Android application by the author 8. This project is characterized by employing a descriptive methodology to design an electronic glove that interprets and translates sign language using Arduino technology and a display interface in an Android application. This project highlights the need to reduce the communication gap.
The work presented by 9, titled "Mobile Application for Learning Literacy with Fitzgerald for Children with Hearing Disabilities," proposes a tablet application as an educational tool, focused on competencies such as concept acquisition and sentence structure, adapted to the child's learning level. This work contributes innovative ideas to facilitate the learning of sign language.
The article "Proposals for Emerging ICT Solutions for People with Disabilities" 10 suggests projects using adapted existing technologies, such as a sign recognition system for the deaf and mute using an interactive camera, applications for post-surgery treatment of cleft lip, mobile interfaces for the blind, and braille self-learning. These advancements significantly contribute to the development of a mobile application for the learning of deaf and mute individuals, facilitating their communication and inclusion.
The main pillar to develop and implement technological tools focused on people with special needs or disabilities will be in response to society, which generates the need to know and know what options exist to facilitate the adaptation and evolution of this population with the use of these tools.
Given the above, it is relevant to mention the existence of different tools for mobile devices in app stores such as: Google Play Store, where there are a variety of applications that help learning and translating sign language for deaf and deaf people who do not know this language, the following comparative table shows the most relevant applications. Fig 1.
Deaf Help. It is an app that has two functions, it is based on converting voice audio to text and text written in spoken voice. This is a mobile tool that makes use of the resources of the Google engine, to allow the interpretation of audio to text 11.
LSApp. It is a mobile tool that aims to improve the quality of life of deaf people, through the learning of sign language. This tool has features such as, sign finder, games to facilitate practice and tips that allow correct communication with deaf people. The one tool is not only focused on learning deaf people, but is also open to the public, allowing the inclusion of people in the learning process, something that is characteristic and contributes to the development of the current project 12.
Speak up. It is a project that provides the opportunity to perform collaboratively, and is born as a solution to allow establishing a dialogue between people with difficulties in their communication. It is a tool with different commercial proposals, with ways to impact banks, hotels, premises or public places providing training and allowing people to become part of their community 13.
Methodology
For the construction of any proposal, a series of steps must be followed, in this sense, the development of the tool called SinSeñas proposed in this article, was carried out using the XP methodology (eXtreme Programming). Where its main objective was always to understand the needs of the client, estimate the effort and create the comprehensive solution that would meet the expectations of the target population 14. It is important to mention that this whole process was articulated under the good practices of project management based on the PMBOOK guide, with the purpose of managing the different areas of knowledge that it comprises. Below are the phases in which the project was developed.
Project phases
During the development of the phases of the project and thanks to the structure of breakdown of activities it was possible to identify the preliminary scope of the project, concept of the problem by presenting different basic functions of the application by means of an analysis of requirements that meet the solution of the problem, a verification of feasibility of the problem and adjustments to the necessary characteristics, through the following activities:
Feasibility study.
Constitution Act.
Technical Offer.
Management Offer.
Economic Offer.
Phase 3.1.1: Planning Phase. In this phase, the different guidelines and plans that were taken into account during the development of the software tool were defined.
Starting from the creation of user stories that allow to realize the functionalities and/or activities of the Project.
An iteration plan is made that evidences the processes that must be executed during a period or term, and the assignment of roles within the project.
Phase 3.1.2: Design phase. A project specification plan is developed, making previous designs and taking into account the functional requirements that allow visualizing the needs of the client and objectives proposed through the following items:
A sequence diagram is established to visualize a logical order of the functionality of the processes.
The preliminary design of mockups is done with the Figma tool, to structure the application and visualize the learning and translation modules, as shown in Fig. 2.
Prototypes are designed where user interfaces are displayed.
Phase 3.1.3: Development Phase. In this phase the iteration plan is developed according to specifications, requirements and planning previously carried out. Different tools are used for the development phase:
MediaPipe Hands: Open-source Google Library enabled high-fidelity hand and finger tracking, which employs machine learning, to infer 21 3D reference points from a hand from a single frame, as shown in Fig. 3. The detection process performed by the library is: first, train a palm detector instead of a hand detector, since estimating bounding boxes of rigid objects such as palms and fists is significantly simpler than detecting hands with articulated fingers. Second, it uses an encoder-decoder feature extractor for greater awareness of the scene context, even for small objects (similar to the RetinaNet approach). Finally, with the above techniques, an average accuracy of 95.7% in palm detection is achieved 15.
Java and Android Studio: We chose to use the Java language in the Android Studio development environment, since the development team was familiar with the language and tool, reducing the learning curve.
SqLite and Firebase realtime: With the use of java in Android Studio, it makes SqLite, together with Firebase, the tools to manage the resources and information of the application database, for its popularity and handling.
Source: Hands [Internet]. mediapipe. [cited 8 June 2022]. Disponible en: https://google.github.io/mediapipe/solutions/hands.html
Figure 3 Hand reference points
Phase 3.1.4: Testing Phase. This phase aims to analyze and evaluate the behavior and performance of the application according to the parameters established based on the defined objectives:
A design of test cases is made, where it is possible to make the previous configuration and requirements that guarantee the correct execution of the tool.
Following the approval of the test cases the execution of these is performed, leaving in evidence the correct functioning.
Subsequently, an acceptance test was performed on interested users who voluntarily agreed to test and give their respective feedback about the application.
After completing the phases that were proposed for the methodological development, where the planning, design, development and testing of the project is exposed, the construction of a software product called SinSeñas is achieved, which meets the needs and requirements of people with a hearing impairment, and thus, improving the condition of social and communicative inclusion.
Results and discussion
After completing the phases of development of the SinSeñas tool, following the methodology where the planning, design and development of the project is exposed, the construction of a learning tool and translation of sign language is achieved, which gives people who use sign language the opportunity to communicate more easily with people who do not know sign language.
During the development of the SinSeñas tool, an analysis was made that allowed visualizing the scope of the project, resources that were available and accessible free software tools, which allowed the agile development of the application such as the programming language Java, IDE Android Studio, Firebase, SqLite and in conjunction with the MediaPipe library, which guarantees an average accuracy of 95.7% in palm detection. In addition, it was possible to highlight different factors that limit development such as storage capacity, the need to use databases in the cloud, artificial intelligence technologies for identification and comparison of information.
As a result you can find that the tool allows in a simple way, translate letters of the alphabet starting from the identification of the hand. Also, it was possible to provide modules with learning materials, images and text that reveal different forms of expression within sign language. It should be noted that both the material oflearning and the translation function were worked with the Colombian fingerprint alphabet.
Thanks to this, it was necessary to carry out test phases, where users who do not know sign language were able to access the information and allow the use of the translator for sign recognition, as shown in the following Figures 4, 5 and 6. Furthermore, It is important to define a scope appropriate to the volume of information and users who can use the tool. This will allow us to improve processing performance, reduce the minimum storage capacity and deploy innovative functions in the technological area, making use of Machine Learning and artificial intelligence technologies, to facilitate the operation and comparison of information in real time.
Conclusions
In the course of the development of the application, it was necessary to test different libraries for the recognition of the gestures captured by the camera, this is due to the complexity required to layout the hand and also due to the provision of the time required to perform the recognition between sign language according to the captured gesture. In this process, some peculiarities were presented that limited the development of functions in the identification and translations of signs due to the large volume of information that is being iterated. Constantly, this represents a large consumption of local storage. Therefore, the use of cloud storage with services such as Firebase real time was considered, both for the storage of signs and learning material.
Also, it is important to mention that the selection of the programming language, libraries and resources that would allow the development of a mobile tool was necessary due to the different limitations that exist around this topic. This is why, to allow a reduction in the learning curve in new technologies, tools with which previous experience was available are used, managing to maintain greater consistency in the activities enshrined in the project development methodology.
Another fundamental aspect is that the guidelines for Colombian sign language were taken into account, this being national intangible, cultural and linguistic heritage. Therefore, the development of the tool is carried out based on the different sources of knowledge of sign language nationally recognized by Colombia (LSC).
In the identification process, the need to use artificial intelligence technologies as a future projection of the scope is present, which allows training algorithms that identify and compare the results of the analysis, in order to provide a response with less margin of error. In the development of the tool, databases are used where the different instructions for the required sign translation are housed. This, in turn, identifies the vectors of key points in the hand, makes a comparison with the information provided and established. in the database, allowing a response to the signal captured by the camera. In the real-time identification process, the constant iteration of Media Pipe resources, generate a performance decrease in the fluidity of the camera. That is why different traces were cut out, such as the visualization of points and indexes on each phalanx of the fingers, which the tool recognizes; The mapping of the hand and fingers with linear traces is then left, allowing, in turn, an improvement in performance.
In the implementation of a mobile tool that allows the translation of signs in real time, it is necessary to highlight that a reduction in the communication gap is achieved, providing not only tools for learning, but also the possibility of communicating at any time. By virtue of this, it becomes essential to reach all audiences interested not only in learning sign language, but also to promote different communication relationships with people who use this language as their first language, also allowing comprehensive inclusion in communication. Colombian sign language.
In addition, it is planned to develop a socialization schedule for the project aimed at reaching a wide audience, especially those people who use sign language.
