Enhancing human-machine interaction: a novel approach to emotion-controlled speech-to-animation

This paper presents a novel framework for emotion-controlled speech-to-animation, addressing the issue of emotional mismatches between speech and facial expressions in existing methods. Our approach synchronises emotional expression across audio and facial animations using State-of-the-Art (SOTA) pretrained models, eliminating the need for costly custom training while ensuring adaptability. A key contribution of our framework is the creation of a novel a Speech-to-Speech (S2S) pipeline for emotional control over generated speech. In addition, we introduce a novel evaluation metric, the Emotion Distribution Divergence (EDD), to assess our models ability to modify the emotions in the original videos. Experimental results demonstrate significant improvements in emotional expressiveness and realism over existing methods, establishing our approach as a major advancement in human-machine interaction, virtual assistants, and emotion-aware IoT applications.