Conference paper

COMPUTER-AIDED PSYCHOTHERAPY BASED ON MULTIMODAL ELICITATION, ESTIMATION AND REGULATION OF EMOTION

Krešimir Ćosić ; University of Zagreb Faculty of Electrical Engineering and Computing, Croatia
Siniša Popović orcid.org/0000-0001-6026-9261 ; University of Zagreb Faculty of Electrical Engineering and Computing, Croatia
Marko Horvat ; University of Zagreb Faculty of Electrical Engineering and Computing, Croatia
Davor Kukolja ; University of Zagreb Faculty of Electrical Engineering and Computing, Croatia
Branimir Dropuljić ; University of Zagreb Faculty of Electrical Engineering and Computing, Croatia
Bernard Kovač ; University of Zagreb Faculty of Electrical Engineering and Computing, Croatia
Miro Jakovljević ; University Hospital Centre Zagreb, Department of Psychiatry, Croatia

Abstract

Contemporary psychiatry is looking at affective sciences to understand human behavior, cognition and the mind in health and
disease. Since it has been recognized that emotions have a pivotal role for the human mind, an ever increasing number of
laboratories and research centers are interested in affective sciences, affective neuroscience, affective psychology and affective
psychopathology. Therefore, this paper presents multidisciplinary research results of Laboratory for Interactive Simulation System
at Faculty of Electrical Engineering and Computing, University of Zagreb in the stress resilience. Patient’s distortion in emotional
processing of multimodal input stimuli is predominantly consequence of his/her cognitive deficit which is result of their individual
mental health disorders. These emotional distortions in patient’s multimodal physiological, facial, acoustic, and linguistic features
related to presented stimulation can be used as indicator of patient’s mental illness. Real-time processing and analysis of patient’s
multimodal response related to annotated input stimuli is based on appropriate machine learning methods from computer science.
Comprehensive longitudinal multimodal analysis of patient’s emotion, mood, feelings, attention, motivation, decision-making, and
working memory in synchronization with multimodal stimuli provides extremely valuable big database for data mining, machine
learning and machine reasoning. Presented multimedia stimuli sequence includes personalized images, movies and sounds, as well
as semantically congruent narratives. Simultaneously, with stimuli presentation patient provides subjective emotional ratings of
presented stimuli in terms of subjective units of discomfort/distress, discrete emotions, or valence and arousal. These subjective
emotional ratings of input stimuli and corresponding physiological, speech, and facial output features provides enough information
for evaluation of patient’s cognitive appraisal deficit. Aggregated real-time visualization of this information provides valuable
assistance in patient mental state diagnostics enabling therapist deeper and broader insights into dynamics and progress of the
psychotherapy.

Keywords

emotion elicitation - multimodal stimulation - cognitive appraisal - multimodal features - emotion estimation – physiological – acoustic – linguistic – facial features

Hrčak ID:

161220

URI

https://hrcak.srce.hr/161220

Publication date:

17.9.2013.

Visits: 1.102 *