Translating is difficult work, the more so the further two languages are from one another. French to Spanish? Not a problem. Ancient Greek to Esperanto? Considerably harder. But sign language is a unique case, and translating it uniquely difficult, because it is fundamentally different from spoken and written languages. All the same, SignAll has been working hard for years to make accurate, real-time machine translation of ASL a reality.
One would think that with all the advances in AI and computer vision happening right now, a problem as interesting and beneficial to solve as this would be under siege by the best of the best. Even thinking about it from a cynical market-expansion point of view, an Echo or TV that understands sign language could attract millions of new (and very thankful) customers.
Unfortunately, that doesn’t seem to be the case — which leaves it to small companies like Budapest-based SignAll to do the hard work that benefits this underserved group. And it turns out that translating sign language in real time is even more complicated than it sounds.
CEO Zsolt Robotka and chief R&D officer Márton Kajtár were exhibiting this year at CES, where I talked with them about the company, the challenges they were taking on and how they expect the field to evolve. (I’m glad to see the company was also at Disrupt SF in 2016, though I missed them then.)
Perhaps the most interesting thing to me about the whole business is how interesting and complex the problem is that they are attempting to solve.
“It’s multi-channel communication; it’s really not just about shapes or hand movements,” explained Robotka. “If you really want to translate sign language, you need to track the entire upper body and facial expressions — that makes the computer vision part very challenging.”
Right off the bat that’s a difficult ask, since that’s a huge volume in which to track subtle movement. The setup right now uses a Kinect 2 more or less at center and three RGB cameras positioned a foot or two out. The system must reconfigure itself for each new user, since just as everyone speaks a bit differently, all ASL users sign differently.
“We need this complex configuration because then we can work around the lack of resolution, both time and spatial (i.e. refresh rate and number of pixels), by having different points of view,” said Kajtár. “You can have quite complex finger configurations, and the traditional methods of skeletonizing the hand don’t work because they occlude each other. So we’re using the side cameras to resolve occlusion.”
As if that wasn’t enough, facial expressions and slight variations in gestures also inform what is being said, for example adding emotion or indicating a direction. And then there’s the fact that sign language is fundamentally different from English or any other common spoken language. This isn’t transcription — it’s full-on translation.
“The nature of the language is continuous signing. That makes it hard to tell when one sign ends and another begins,” Robotka said. “But it’s also a very different language; you can’t translate word by word, recognizing them from a vocabulary.”
SignAll’s system works with complete sentences, not just individual words presented sequentially. A system that just takes down and translates one sign after another (limited versions of which exist) would be liable to creating misinterpretations or overly simplistic representations of what was said. While that might be fine for simple things like asking directions, real meaningful communication has layers of complexity that must be detected and accurately reproduced.
Somewhere between those two options is what SignAll is targeting for its first public pilot of the system, at Gallaudet University. This Washington, D.C. school for the deaf is renovating its welcome center, and SignAll will be installing a translation booth there so that hearing people can interact with deaf staff there.
It’s a good opportunity to test this, Robotka said, since usually the information deficit is the other way around: a deaf person who needs information from a hearing person. Visitors who can’t sign can speak, and the query can be turned to text (unless the staff member can read lips) and responded to with signs, which are then translated back into text or synthesized speech.
It sounds complicated, and in a technical way it is, but really neither person needs to do anything but communicate the way they normally do, and they can be understood by the other. When you think about it, that’s pretty amazing.
To prepare for the pilot, SignAll and Gallaudet worked together to create a database of signs specific to the application at hand or local to the university itself. There’s no comprehensive 3D representation of all signs, if that’s even possible, so for now the system will cater to the environment in which it is deployed, with domain-specific gestures being added to a database on a rolling basis.
“That was a huge effort, to collect the 3D data of all these signs. We just finished, with their support,” said Robotka. “We did interviews, collected some conversations that occurred there, to make sure we have all the language elements and signs. We expect to do that kind of customization work for the first couple of pilots.”
This long-running project is a sobering reminder of both the possibilities and limitations of technology. True, automatic translation of sign language is a goal only just becoming possible with advances in computer vision, machine learning and imaging. But unlike many other translation or CV tasks, it requires a great deal of human input at every step, not just to achieve basic accuracy, but to ensure the humanitarian aspects are present, as well.
After all, this isn’t just about the convenience of reading a foreign news article or communicating abroad, but of a class of people who are fundamentally excluded from what most people think of as in-person communication — speech. To improve their lot is worth waiting for.
Comment