论文标题
顺序到序列预测模型：从韵律到交流手势
Sequence-to-Sequence Predictive Model: From Prosody To Communicative Gestures
论文作者
论文摘要
交流手势和语音声音紧密相连。我们的目标是根据声学预测手势的时机。也就是说，我们想预测何时发生某种手势。我们开发了一个基于具有注意机制的复发神经网络的模型。该模型在自然二元相互作用的语料库中进行了训练，其中言语声音和手势阶段和类型被注释了。该模型的输入是一系列语音声音，输出是一系列手势类别。我们用于模型输出的类是基于手势阶段和手势类型的组合。我们使用序列比较技术来评估模型性能。我们发现，该模型比其他模型可以预测更好的某些手势类别。我们还进行消融研究，该研究表明基本频率是手势预测任务的相关特征。在另一个子体验中，我们发现包括眉毛动作为节拍手势，可以提高表现。此外，我们还发现，经过培训的一个给定发言人数据的模型也适用于同一对话的另一个演讲者。我们还执行一个主观实验，以衡量受访者如何判断虚拟药物的自然性，时间一致性和语义一致性。我们的受访者对模型的产出进行了有利的评价。
Communicative gestures and speech acoustic are tightly linked. Our objective is to predict the timing of gestures according to the acoustic. That is, we want to predict when a certain gesture occurs. We develop a model based on a recurrent neural network with attention mechanism. The model is trained on a corpus of natural dyadic interaction where the speech acoustic and the gesture phases and types have been annotated. The input of the model is a sequence of speech acoustic and the output is a sequence of gesture classes. The classes we are using for the model output is based on a combination of gesture phases and gesture types. We use a sequence comparison technique to evaluate the model performance. We find that the model can predict better certain gesture classes than others. We also perform ablation studies which reveal that fundamental frequency is a relevant feature for gesture prediction task. In another sub-experiment, we find that including eyebrow movements as acting as beat gesture improves the performance. Besides, we also find that a model trained on the data of one given speaker also works for the other speaker of the same conversation. We also perform a subjective experiment to measure how respondents judge the naturalness, the time consistency, and the semantic consistency of the generated gesture timing of a virtual agent. Our respondents rate the output of our model favorably.
