Original scientific paper
https://doi.org/10.24138/jcomss.v14i1.410
Towards a Synesthesia Laboratory: Real-time Localization and Visualization of a Sound Source for Virtual Reality Applications
Ahmet Kose
; Tallinn University of Technology, Estonia
Aleksei Tepljakov
; Tallinn University of Technology, Estonia
Sergei Astapov
; Tallinn University of Technology, Estonia
Dirk Draheim
; Tallinn University of Technology, Estonia
Eduard Petlenkov
; Tallinn University of Technology, Estonia
Kristina Vassiljeva
; Tallinn University of Technology, Estonia
Abstract
In this paper, we present our findings related to the problem of localization and visualization of a sound source placed in the same room as the listener. The particular effect that we aim to investigate is called synesthesia—the act of experiencing one sense modality as another, e.g., a person may vividly experience flashes of colors when listening to a series of sounds. Towards that end, we apply a series of recently developed methods for detecting sound source in a three-dimensional space around the listener.We also apply a Kalman filter to smooth out the perceived motion. Further, we transform the audio signal into a series of visual shapes, such that the size of each shape is determined by theloudness of the sound source, and its color is determined by the dominant spectral component of the sound. The developed prototype is verified in real time. The prototype configuration is described and some initial experimental results are reported and discussed. Some ideas for further development are also presented.
Keywords
virtual reality; synesthesia; acoustic localization; microphone array; therapy
Hrčak ID:
199217
URI
Publication date:
15.3.2018.
Visits: 1.896 *