Do you trust an avatar to send your client a sign language message?

We live in the IA area. Algorithms are getting increasingly more sophisticated and performing tasks once reserved for humans. They don’t get tired, don’t ask for a raise, blah, blah.  On many occasions, IA does things humans just can’t.  While in others, it is just pure old fashion capitalism: to produce more with less.

Avatars as sign language interpreters threaten people’s jobs. They are much easier, faster, cheaper, and more convenient to use. The programmer calls a web service (or like), passing the text to be transformed into sign language, and suddenly, a humanoid figure starts exhibiting some embodied behavior. The programmer can even choose the avatar's appearance: gender, race, skin color, etc. Piece of cake. What else can one ask for?! —  I would ask for assurance that the right message is being delivered.

Let’s take, for instance, the terms of use everyone has to agree with to start using an online service. You ask the juridical team to craft the texts. It is supposed to be legally binding - critical stuff -   Once it is ready to embed into your app, you decide to be inclusive and add an online avatar to interpret the terms. How can you check whether it is delivering the message appropriately?! What if the IA fails (and yes, it can fail) and provides the wrong message?!

In a recent paper (December 2022), Ismashev and colleagues wrote: "Research into sign language recognition has demonstrated promising potential for robust automatic sign language recognition. However, the area of sign language synthesis is still in its infancy.”  Several other research papers report the extensive use of fingerspelling, which harms the user experience. 

Many online sign language synthesizer services could come swinging at this argument, claiming that their IA machines implement such and such strategies and have great accuracy. Accuracy – that can be deceiving. What is a good level of accuracy?  Well, that will depend on the purpose of your IA. If you’re processing images getting motorcycle license plates in a busy city is one thing. Building a piece of technology to mediate your client's digital experience is an entirely different ball game. Humans tolerate very little noise during the interaction. They quickly get frustrated. The rule is simple:  energy expenditure. The best user experience demands less energy from humans to accomplish their real-world objectives. Sign language synthesizer shortcomings will have to be compensated (when possible) by your client’s effort - energy expenditure.

It is not that we are against the use of avatars to convey sign language content. We’ve been involved in several research projects in that regard. What’s in debate here is the synthetically generated avatar’s embodied behavior, especially when no domain constraints exist - the web service’s “big mouth” would process any text it swallows no matter the context. How would a programmer, not versed in sign language, be assured the avatar’s signaling is reliable?

To tackle this, we ask for the help of real-world sign language interpreters – professionals. To begin with, human interpreters will definitively fire your customer’s language acquisition device along with their mirror neurons. That comes naturally when you involve humans in delivering a message to other humans. Okay, nice, but how can a programmer ask for such help? Or, how can a programmer even know when, during the interaction, it is appropriate to convey a message through sign language? After all, people with hearing disabilities interact with computers.

Another significant value added by human interpreters is empathy. Human interpreters can create an emotional connection with other humans, which is crucial for more efficient communication and the psychological well-being of people with hearing disabilities. Human interpreters also explain complex concepts better and can add information, improving message conveyance. 

At FFIT, we’ve developed a gallery of accessible micro front-ends (MFEs) - live components - ready to be embedded into your app. A micro front-end encompasses a user micro journey. So, if, in that particular interaction, a sign language message is in order, a video placeholder is available at the appropriate point. These MFEs are also available at design time via our exclusive Figma plugin. FFIT’s clients have access to interpreters who can record these videos.

FFIT: Humanity, Productivity.