TED-Culture: Culturally Inclusive Co-Speech Gesture Generation for Embodied Social Agents

🌀 Abstract

Generating natural and expressive co-speech gestures for conversational virtual agents and social robots is crucial for enhancing their acceptability and usability in real-world contexts. However, this task is complicated by strong cultural and linguistic influences on gesture patterns, exacerbated by the limited availability of cross-cultural co-speech gesture datasets. To address this gap, we introduce the TED-Culture Dataset, a novel dataset derived from TED talks, designed to enable cross-cultural gesture generation based on linguistic cues. We propose a generative model based on the Stable Diffusion architecture, which we evaluate on both the TED-Expressive Dataset and the TED-Culture Dataset. The model is further implemented on the NAO robot to assess real-time performance. Our model surpasses state-of-the-art baselines in gesture naturalness and exhibits rapid convergence across languages, specifically Indonesian, Japanese, and Italian. Objective and subjective evaluations confirm improvements in communicative effectiveness. Notably, results reveal that individuals are more critical of gestures in their native language, expecting higher generative performance in familiar linguistic contexts. By releasing the TED-Culture Dataset, we facilitate future research on multilingual gesture generation for embodied agents. The study underscores the importance of cultural and linguistic adaptation in co-speech gesture synthesis, with implications for human-robot interaction design.

🌀 Objective Evaluation

The Quantitative Results on TED-Culture for each specific language and merged, as well as TED-Expressive. The results are the average of five experimental tests. We compare the proposed diffusion-based method against recent state-of-the-art (SOTA) methods and ground truth. Lower values are better for FGD, while higher values are better for the other metrics.

Results for TED-Culture

A. Indonesian

B. Japanese

C. German

D. Italian

E. French

F. Turkish

G. Merged

Results for TED-Expressive

🌀 Subjective Evaluation

Demographic

Histogram of Gender

Histogram of Age

Language Acquisition Stats

Auxiliary Information about New Technology, Video Games and Music

Model Performance

User Study Results. The ratings for motion naturalness, smoothness, and synchrony are assessed on a scale of 1 to 5, where 5 indicates the highest performance. All the results in the table are presented in the format of Average (SD).

The correlation between language acquisition and the cultural back-ground of the participants. The ratings for motion naturalness, smoothness, and synchrony are assessed on a scale of 1 to 5, where 5 indicates the highest performance. All the participants’ first or secondary proficient language is Indonesian. All the results in the table are presented in the format of Average (SD).

🌀 Ablation Experiments

A.Model Structure Experiments

The Quantitative Results on TED-Culture Merged, focusing on model structure. The results are the average of five experimental tests. We compared with different model architectures and conducted ablation studies to identify the relatively optimal model structure for this dataset. Lower values are better for FGD, while higher values are better for the other metrics.

B.Proposed Modules Experiments

Ablation Study on the Proposed Modules Based on TED-Culture Merged. The results are the average of five experimental tests. We investigate the effectiveness of the proposed modules: the Diffusion Gesture Stabilizer and implicit classifier-free guidance. Lower values are better for FGD, while higher values are better for the other metrics.

C.Text Embedding Experiments

Ablation study on the impact of using corresponding text embeddings in the DiffCulture model after incorporating the text modality on TED-Culture French. The results represent the average of five experimental tests. We investigate the impact that whether or not using the corresponding language of FastText Word Embedding can have on model (with text modality) performance. For FGD, lower values indicate better performance, while higher values are preferable for the other metrics.

🌀 Other Links​

TED_Culture_Dataset: https://github.com/Yixin-Shen-1218/TED_Culture_Dataset

TED_Culture_Visualizer: https://github.com/Yixin-Shen-1218/TED_Culture_Visualizer

NAO_Gesture_Generation: https://github.com/Yixin-Shen-1218/NAO_Gesture_Generation

Virtual Agent Stimuli: https://www.youtube.com/playlist?list=PLTTEUXLjZvB9-ViI1ZkmpZdphsQb92cAt

Social Robot Stimuli: https://www.youtube.com/playlist?list=PLTTEUXLjZvB8qEKfwHjQIgOJ3ahP_gOPS