Hala is a bi-lingual (Arabic and English) robot receptionist located at Carnegie Mellon in Qatar. Hala is presented to users as a 3-D animated face on a screen. Users type to her, and she replies in speech synced with realistic facial expressions. However there are two existing problems with the robot. First, Hala's animation engine does not fully adhere to existing research on face dynamics, which makes it difficult to create natural and interesting facial expressions. Natural expressions help towards an engaging user experience by articulating non-verbal aspects (ex: confusion, glee, horror, etc). Second, while speaking in Arabic lip-movements are not realistic because they were adopted from English utterances. In this work we address these two limitations.

Similar to the movie and video-game industry, we leverage Paul Ekman's seminal work on Facial Action Coding System (FACS) to demarcate Hala's 3D face model into muscle-primitives. These primitives are used to compose complex, yet natural, facial expressions. We have also authored an in-house tool, which allows non-programmers (for ex: artists) to manipulate the face in real-time to create expressions.

The sounds humans make while talking are symbolically captured as “phonemes”. The corresponding shapes of the lips, for these sounds (i.e. phonemes), are called “visemes”. We used existing research and observed each other (and a mirror), to develop visemes that accurately capture Arabic pronunciations. Hala can thus utilize English and Arabic visemes for accurate lip-movement and syncing. We empirically tested and evaluated our work by comparing it with previous lip movements for common Arabic utterances. Nonetheless, certain pronunciations can fire less-than-ideal visemes if they are preceded by silence.

Upon identifying and addressing these limitations, Hala has 11 new facial expressions for a more natural looking and behaving robot. This work also pioneered the first implemented subset of Arabic visemes on a robot.


Article metrics loading...

Loading full text...

Full text loading...

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error