Original scientific paper

https://doi.org/10.17559/TV-20240919001998

Study on Multimodal Session Recognition Based on Graph Neural Networks

Zhixue Wang ; School of Mathematics and Computer Science, Ningxia Normal University, Guyuan, 756000, Ningxia, China *
Hongwu Zhang ; School of Mathematics and Computer Science, Ningxia Normal University, Guyuan, 756000, Ningxia, China
Kai Kang ; School of Mathematics and Computer Science, Ningxia Normal University, Guyuan, 756000, Ningxia, China

* Corresponding author.

Abstract

As one of the special research directions of text emotion recognition task, the conversation emotion recognition task needs to take the conversation context, speaker and other factors into account to accurately identify the emotion state. To address challenges in multimodal information fusion and global-local feature extraction, this paper proposes a multimodal session recognition method based on multilevel attention mechanism and multistream graph neural network. Firstly, two key issues of the multimodal session recognition problem are analysed; secondly, a multimodal session emotion recognition model based on the improved attention mechanism strategy and the improved graph neural network is proposed to address the two issues; and finally, the effectiveness of the proposed method is verified through the analysis of simulation experiments. The simulation results show that the method achieves good accuracy in IEMOCAP and MOSEI datasets and effectively improves the efficiency of session fusion recognition.

Keywords

attention mechanisms; graph neural networks; information fusion; multimodal session recognition

Hrčak ID:

330537

URI

https://hrcak.srce.hr/330537

Publication date:

1.5.2025.

Visits: 434 *