Original scientific paper
https://doi.org/10.30924/mjcmi.31.1.6
From news to charts: rethinking market prediction with multimodal AI
Haochen Guo
; Southeast University, School of Economics and Management, Nanjing, China; Laboratory of Philosophy and Social Sciences at Universities in Jiangsu Province-Fintech and Big Data Laboratory of Southeast University, Nanjing, China
Petr Polak
; Mendel University in Brno, Faculty of Business and Economics, Brno, Czech Republic
*
* Corresponding author.
Abstract
In the age of data-driven finance, the proliferation of multimodal information—ranging from news articles to technical charts—has challenged the adequacy of traditional unimodal predictive models. This study introduces a novel multimodal AI framework that integrates textual and visual signals to forecast financial market movements. Grounded in behavioral finance theory, the model combines transformer-based language encoders and CNN-based image processors through attention-guided fusion, allowing it to simulate how investors cognitively process and react to complex information environments. A case study of the 2023 Silicon Valley Bank collapse demonstrates the model’s superior predictive performance and its ability to identify predictive patterns that are behaviorally interpretable and consistent with established cognitive bias theories, such as loss-aversion-type asymmetry and herding-like clustering. The paper also critically examines the ethical and regulatory implications of deploying such systems, emphasizing the need for explainability, behavioral neutrality, and inclusive oversight. By bridging algorithmic forecasting with social cognition, this research rethinks the role of AI in shaping financial knowledge and behavior.
Keywords
Multimodal AI; Financial forecasting; Behavioral finance; Sentiment analysis; Technical charts; Explainable AI; Cognitive bias; Algorithmic governance
Hrčak ID:
348355
URI
Publication date:
25.2.2026.
Visits: 0 *