Tools and Methods Applied in Interactive Systems to Evaluate the User Experience With Deaf/Hard of Hearing Children

The evaluation of a system is part of the development process that must be done preferably with potential users, since they can give new ideas and relevant feedback about the user experience of it, so a User-Centered Design approach could give better results to the final product. But involving users with special needs like those in condition of disability is not an easy task for researchers, since there are variables that must be carefully taken into account in order to successfully extract information from these special users. In this study, we applied different evaluation methods in the design of an Interactive System where Deaf and Hard of Hearing children were part of it. Some of the methods used were modified to satisfy the needs of the children as well as their skills.


INTRODUCTION
An Interactive System (IS) is any system based on computers that engages in ongoing interaction with one or more human users [1]. Examples of IS are: a digital photo album, robots, video games or mobile devices among many others and can be used in different contexts like entertainment, education, health, military or marketing. In an educational context, IS are intentionally designed to teach material traditionally taught within an academic environment, one example of such systems is Serious Games (SG) [2] which are a type of games that are not designed just for pure entertainment, instead, their purpose is to change behaviors and impart knowledge [3]. Technology mediation has evidently become the key driver in development of learning [4] even for persons with some sort of disability like Deaf or Hard of Hearing (DHH) people.
Part of the development process of this kind of systems involves an evaluation phase where they are tested either by experts or end-users. When IS are designed for some specific users, they should be considered during the evaluation process in order to have a product that satisfies their needs. Unfortunately, not all users participate in this kind of activities, for instance, people with disabilities are not included in such tasks, mainly because traditional evaluation techniques do not always work with them, and this is particularly true when these special users are children.
This study is focused on the evaluation of Interactive Systems and how DHH children can be involved during this process. A case study was carried out with DHH children for the design of an interactive game.
The paper is structured as follows. Section 2 gives an overview about DHH children and their educational challenges as well as a brief introduction to Interactive Systems and evaluation methods that has been used with them. In section 3 the case study is presented. The discussion about the findings of the case study are presented in section 4. Section 5 concludes this study and future work is proposed.

Education of Deaf and Hard of Hearing Children
All children should be exposed to a first language during the first 5 years of age [5] [6] and for DHH children it should be a sign language (SL) used or accepted in their countries, even if they are going to receive a cochlear implant [6]. This is important to get the necessary skills to learn a second language in a written form [7] or get access to knowledge in other areas like math and sciences [8]. Unfortunately, 90% of DHH children cannot get access to a sign language during this period due to they are born to hearing parents [6][9] [10], which means that they probably do not know any SL and thus they cannot communicate with their children. This late acquisition of a first language will delay the learning process of their second language and this will lead to poor literacy skills and leave them in a clear disadvantage with their hearing peers [11]. DHH children can benefit from fitting a cochlear implant, an electronic device that replaces the function of the damaged inner ear to provide sound signals to the brain [12]. Children with cochlear implants can receive auditory-verbal therapy, where they must listen and identify sounds to learn an oral language [12][13].

Interactive Systems
Nowadays, Interactive Systems can be used as a powerful educational resource because of its engaging characteristics that attracts people to learn [14]. In [15], Interactive Systems are defined as a user-oriented field of study that focuses on the communication relationship between the user and the system. The interactivity level is acquired by the existing resources allowing the user to establish a participation and communication process with the system.
The design of an interactive system for children should follow a development process called User-centered design (UCD) which is a process that analyzes and identifies user's needs. UCD involves a set of stages such as: identify needs, analyze data, design prototype, implementation and evaluation.
In the analysis stage where the interest is to get to know the user's needs, Cano et al. [16] proposed a model where different aspects were analyzed with children with impaired hearing, children with cochlear implant and those whose communication is through sign language.

Evaluation Methods + DHH Children
DHH children are rarely taken into account in technology development, despite the potential technology that is currently available to aid them in language acquisition [17]. In order to involve children with hearing impairments in technology development, it is important to understand that deaf community has very different cultural-linguistic traditions and personal background [18], this means that evaluation techniques or tools that are usually used with hearing children may not work with deaf children, since they have diverse forms to study, work and interact [19].
Some studies show different approaches when evaluating systems with DHH children; Egusa et al. in [20] conducted an experiment to evaluate whether the participation of DHH children in a story told by means of an interactive puppet theater could ensure an experience that they could enjoy. Children participated by using physical movements in front of a kinect device to define the flow of the story. The evaluation in this study was based on a survey and a five-point scale (from strongly agree to strongly disagree) for every question. This type of evaluation is one of the most common due to its simplicity. Other studies [21] [27] have used Likert-scale surveys to get information out of the evaluation process with DHH children.
In [28], E-Drawings were used as an evaluation method for experiments with deaf children. The experiment gave children the opportunity to use a drawing program (TuxPaint) instead of paper and felt tips to illustrate a story they previously read, and thus, the level of reading comprehension could be measured.
A different approach was employed by Zainuddin et al. in [29] where they carried out a heuristic evaluation on a courseware for the deaf, which means that end-users were not part of the evaluation process, instead, several experts from different fields were involved in it. The reason given by the authors of the study to use this kind of evaluation was that it can help identify more problems than evaluating just with users.
According to Ornella in [30], using traditional methods (observation, thinking aloud, questionnaires) to evaluate software applications with DHH children, requires to tune such methods to these special users. That's why, some guidelines were proposed in order to support this tuning. The necessary information to create the guidelines was obtained from people who work or live with the children and the personal experience of the researcher. This study concludes that the following recommendations should be considered when software is being evaluated with DHH children: Tests should be carried out in a public space like school and with a group of children; the sessions must last less than 30 minutes and without the presence of the parents; the instructions must be provided both orally and with sign language and if written instructions are necessary, the language used have to be carefully chosen; children should be asked to repeat instructions given; finally, at the end of the test they should give their impressions about it.
DHH children tend to communicate visually due to the problems they have in the auditory channel, therefore, it is important to use visual scales of qualification. Usually, the scales of qualification use facial expressions, where different emotions represent values, for instance, a high value is represented as happiness or a low value as sadness. EMODIANA [31] a visual tool of subjective emotion measurement based on 10 graphic representations of different emotions (affection, joy, satisfaction, shame, sadness, boredom, seriousness, nervousness, surprise, fear) of a character and the intensity of every emotion by means of colors. EMODIANA has been adapted to evaluate videogames with children with down syndrome [32].
A work presented by Cano et al [33] have analyzed different methods for the evaluation of interactive products applied with children. Table 1 shows some of the methods applied and what advantages and disadvantages were found in each evaluation method. Table I. Evaluation methods applied to children with cochlear implant [33].

Method
Advantages Disadvantages Direct observation [34] The children have no need to speak, the expressions and body movements are analyzed.
A shy child who feels he is being watched could not interact with a product in a natural way. Thinking aloud [35] Children can collaborate during the work and be more confident when it is their turn to express their feelings and opinions.
The method can be very challenging for children who do not speak or have a reduced vocabulary.
Drawing intervention [36] It does not require the children to speak since drawings are the outcome to be analyzed.
Drawings are open to wide interpretation, that is why experts are needed to evaluate the results. Picture card [37] Rates usability and UX using visual information based on pictograms.
The children can fail to understand the meaning of the pictograms. Wizard of Oz [38] The child is only required to interact with a product without noticing that s/he is being observed from another room.
The assistant must constantly be attending the child to ensure that they do not lose motivation. aFun toolkit [39] It can be applied with children with low cognitive skills due to it is mainly visual.
Due to the amount of information given to the children, it could be challenging for them. Surveys [40] For children, this type of evaluation can be employed in several ways: child-child, teacher-child, assessor-child.
Questions being asked must be carefully designed and a adequate vocabulary must be used to avoid misinterpretation.
The evaluation methods that have been used with children with cochlear implant have been used like subjective (verbal) and objective (nonverbal). With deaf children, applying subjective methods requires that the evaluator knows sign language. Furthermore, the number of methods suitable to use with children with special needs is smaller compared with those for children with no disabilities.

CASE STUDY
The case study was carried out at the Institute for Blind and Deaf children of Valle del Cauca (INCSVC) in Cali, Colombia. 11 children (three girls and eight boys) with cochlear implants participated with the consent of their parents and their ages ranged from 7 to 11 years old. All participants are profoundly deaf in both ears and receive Auditory-verbal therapy. 5 children are in pre-kindergarten and 6 are in Transition. The case study was also done at Institute of Special Therapy of the Senses (ITES), Club Leones in Cali, Colombia, with 8 children (3 boys and 5 girls) that communicate with sign language. These 8 children are in basic first grade.
The Drawing Intervention method was applied with the children from the both institutions because they are in an early stage of acquiring listening or signing language skills to be able to communicate and their process of acquiring a language is just beginning, that's why a nonverbal method was necessary in order to get information about them.

Figure 1. Drawing Intervention Method at INCSVC with Children of Prekindergarten and Transition.
The purpose of this test was to get relevant aspects about how to design a character that can be comprehensible and likable to them, that is why the test was different for children in pre-kindergarten and transition levels. For every prekindergarten child, a different character was assigned, they start with an initial shape where they must complete parts of the body and color it. At the end, they are asked to give a name to their character, although not all of them did it. This activity, was done with the support of the teacher who explained the activity to the children.

Figure 2. Test with pre-kindergarten children at INCSVC
Children completed the drawings but it was difficult for them to give it a name, since they are in a process of acquisition of new vocabulary. This is the reason why some of them chose words that were familiar to them like "Papá" (Dad in English) and this suggests that they still don't have good communication skills and their vocabulary is limited ( Figure  2). Their imagination is not developed at the same pace of a hearing child, this was evident when they knew that they had to draw imaginary characters but before starting they stared the initial shape for a long period of time because they did not understand what it was and were confused about which part of the body they had to draw. We also noticed that girls were more careful with the drawing and took more time since they usually draw accessories and used girly colors. The objective of the drawing was to know their personalities, motivations, colors and how they are stimulated through drawing.
The same test was carried out with children in transition ( Figure 3), however, there was a better interaction with them, they did give a name to their character and used more colors. With both test we could also involve children in the creation of characters during the whole design process.

Figure 3. Test with children in transition at INCSVC
With the children from ITES, there was another variation of this method. In this case, the child had to make a character by putting together different parts of the body that were cut previously, so the child had the opportunity to choose from different pieces of the same part of the body according to their likes, for instance, there were different kinds of legs, arms, etc. (Figure 4). We also wanted to know about their preferences concerning to colors, so they could freely choose the color of the papers to be used as backgrounds, for instance, boys chose strong colors like blue or brown (figure 4) while girls chose softer colors like pink or purple.

Figure 4. Drawing Intervention Method at ITES with human characters
At ITES, children created 2 different kinds of characters, one of them was based on real people, like kings, princesses, heroes, etc. The other kind of character was created from parts of a non-human body ( Figure 5). They were also asked to draw themselves, where they draw the parts of the body that most characterizes them, like hair style, eyes size or accessories.

Figure 5. Drawing Intervention Method at ITES with nonhuman characters
The Thinking Aloud method was also applied with the children from INVSVC in transition level, but it had to be modified due to their lack of vocabulary and this makes it difficult to extract information from them. So, we decided that based on the characters they made (Figure 3), they had to create a story with each one of the characters. This test was made as a cooperative task between all children, where they had to choose the role of every character (hero, villain, secondary character, among others) and that way they could understand the vocabulary that will be used in the story of the game. This activity was really productive since children worked together and actively, and did not inhibited to give their opinion. The use of technologies was also evaluated with these children since the proposed interactive system to be developed has a physical and technological component. That is why, two methods (Direct Observation and Fun Toolkit) were applied with the children using Speaking with TEO [41], Vivoso [42], Memory Kids [43] and Grupolandia [44]. In this case, children were evaluated individually. These evaluations were made only at INCSVC.
Direct observation was applied while the child interacted with the apps. It was necessary to video record the interactions of the child in order to capture information from two different angles; one camera was located in front of the child to see his/her expressions while using the apps and another camera was behind him/her to capture how the app was being used.
From the Fun Toolkit method, we used the smileyometer which is a discrete Likert type scale composed by faces that show a different expression from Awful (sad face) to Brilliant (happy face) [45]. Figure 7 shows a five-scale smileyometer used in its original version.

Figure 7. Five scale smileyometer [45]
Then, as children got confused by the faces because they did not recognize the 3 expression in between (Not very good, Good, Really Good) we had to create different versions of this smileyometer with faces and expressions that children could recognize more easily (Figure 8).

Figure 8. Variation of five-scale smileyometer
Unfortunately, results with this new smileyometer were not satisfactory for the same reasons.
Some of the tasks that they had to do after using the apps were: -Associate a color with an emotion -Associate an emotion with the experience after using the apps The intention of this associations was to find a way to create a new evaluation scale based on their favorite colors which would be attached to an emotion. Figure 8 shows some of the colors used, so the children had to choose an emotion for every color depending on how much s/he likes every color, but again this was not successful.

Figure 9. Color-based scale
Finally, we decided to use a binary smileyometer where children had to choose either a happy face or a sad face to evaluate the experience with the apps.

Figure 10. Binary-scale smileyometer
Teachers give the same kind of grades to children during regular classes (happy or sad face) so this could be one of the reasons why children got confused with a new grading structure.

DISCUSSION
Qualitative results were gathered from both institutions (INCSVC and ITES) using the Drawing Intervention method, while quantitative results were retrieved only with the 11 children from INCSVC using the remaining methods (Thinking Aloud, Direct Observation, Fun Toolkit).
Basic open questions were also asked to every child from INCSVC, such as: -¿Do you have a tablet, smartphone or PC at home? -¿Do you use the tablet/smartphone/PC alone? -¿Do you use the tablet/smartphone/PC with your parents? -¿How many times a day do you use the tablet/smartphone/PC at home? These specific questions made the children inhibit and they felt shy, which we believe could have changed what they answered, so those who have one or more of the aforementioned devices at home also said that they used it only with their parents. Also, Tablets are more used than PCs when playing games.
It is important to note that these questions had to be adjusted to a comprehensible language for the children with cochlear implants, so most of the questions had to be repeated several times and using different words so they could understand them.
Nowadays, existing evaluation methods are not designed for children in condition of disability, so it is important for researchers to know which ones are suitable or can be modified for a particular disability and which ones are not. It is also important to bear in mind that such modifications must be adjusted according to the needs and skills of the children.
Drawings were used in this case study more as a design resource than an evaluation method, but carrying out this activity once the children have used a system or app, could also give important data to analyze their emotions and thus, get information about the user experience with such system or app. Through drawings, the child can represent not just the conscious mind but also the unconscious mind [46]. The stroke, shapes and colors can tell what the child feels inside, all s/he cannot express verbally. In psychology, drawings are used to analyze personality traits, these are called "Human Figure Drawing Test" (HFDT) proposed by Elizabeth Koppitz [47]. With these kind of drawings, it is possible to extract information from different elements on it according to the age and gender of the child. Such elements are: nose, eyes, head, arms, ears, among others.
We noticed that most of the children with cochlear implants (pre-kindergarten and transition from INCSVC) did not draw their ears or implants. The ones that draw their ears, made them really small. It was curious because other parts of their bodies were drawn with more detail. Pre-kindergarten children had more difficulty to draw their extremities like hands and feet. According to the HFDT, it could mean that these children are shy and with a special education. Children from ITES are more visual due to their primary communication channel is through the sight (they have no implants and use sign language) and their drawings were adequate for every character and colored them based on their likes.
When we applied the smileyometer from the Fun Toolkit method, the emotions we finally used (binary scale) were not the most adequate, since it had only two options: sad (negative) and happy (positive), but the negative emotion was not the right one in some cases because through the Direct Observation method we noticed that children can show an emotion of anger or frustration when the activity was not done correctly, they did not understand it or it was difficult for them, so this does not mean they are sad but angry or frustrated.

CONCLUSIONS AND FUTURE WORK
Designing and evaluating interactive systems with children is important because adults live in a "different world" so they cannot understand what a child feels and wants. This statement is more evident if systems consider children in condition of disability as potential users of it, because it is more difficult to understand how it feels to live without the ability to do something that for most people take for granted.
Communicating with DHH children was not an easy task, especially when the objective was to extract information or find out their opinion about some product or other. It is important to involve not only the expert evaluators, but teachers, a psychologist and speech therapists because they can assist in the acquired knowledge bases of the children and the way of interacting with them.
This study is part of a larger project that is expected to provide a framework for the design of interactive collaborative systems to support literacy teaching to deaf children, that's why in this research we aimed to involve DHH children in the design of an Interactive System and thus identify which evaluation methods can be used or modified to this particular disability for future designs of systems such as serious games.
As future work, collaborative learning strategies will be included to be evaluated with the evaluation methods presented in this study, as well as new methods that can also assess the progress during the learning process of DHH children.