A Hybrid CNN and LSTM based Model for Financial Crisis Prediction

: The detection and prediction of financial crises in listed companies are crucial for investors to mitigate potential losses. Traditional prediction methods primarily rely on financial indicators, yet they often overlook valuable insights hidden in financial text. To address this limitation, our study explores the integration of financial indicators and financial text from annual reports to enhance financial crisis prediction. We propose a two-step approach, leveraging a Convolutional Neural Network (CNN) model to extract features from financial indicators and utilizing a Long Short-Term Memory (LSTM) network with attention mechanism to capture the underlying semantics in financial text. Subsequently, we combine the extracted features from both sources for effective classification. Through extensive experiments with various models, we demonstrate the efficacy of our combined approach in achieving optimal prediction results. Our findings highlight the importance of considering financial text alongside traditional financial indicators for enhanced financial crisis detection and prediction. The proposed methodology contributes to the existing literature and offers valuable insights for investors and financial analysts seeking more accurate and comprehensive risk assessment tools.


INTRODUCTION
Many companies raise capital for expansion by issuing stocks, bonds, and other means after listing on the stock exchange.The listed companies use the raised funds to expand their production and operation through innovation, research and development, investment, and other means.At the same time, investors are worried about investment failure, and listed companies are worried about business failure.Therefore, financial risk control of the enterprise is an unavoidable issue for investors and listed companies.The financial crisis of a company mainly refers to the uncertainties faced by the company in terms of operation, financing, and investment.Specifically, it refers to the unreasonable financial structure of the company and improper financing, which may cause the company to lose its debt solvency and reduce the expected return of investors.Therefore, how to detect and predict the financial crisis of a company is of great importance.An early and precise financial crisis detection and prediction can help companies and investors identify the risk level specific to the business, prioritize those risks, develop ways to avoid them and outline steps to manage them should they happen [1].Financial crises do not happen all of a sudden, but go through a process of gradual evolution, which makes it possible to predict the financial crisis.However, predicting financial crises is very challenging, especially for investors, because investors can only employ the information publicly disclosed by the companies to investigate the likelihood of the company encountering financial crisis, and the information disclosed by listed companies is very limited [2].In this work, we approach the problem from the standpoint of investors; we hope to construct a useful model.By the model we can analyse all annual reports and financial data of companies and it can provide an early warning of financial risks.An annual report is a comprehensive report detailing a company's activities throughout the preceding year.Its purpose is to provide users, such as shareholders or potential investors, with information about the company's operations and financial performance.Most annual reports generally contain the following sections: financial statements, performance highlights from the preceding year, and performance and outlook for future years.Financial statements are a key component of the annual report and provide its users with quantitative data regarding specific aspects of its financial performance in the previous fiscal year.Typical financial statements include balance sheets, income statements, and cash flow statements, each consisting of a set of financial indicators.Annual reports usually dedicate a section to highlighting some of the company's key achievements, such as goals reached, or awards received by the company or its employees.What is more, annual reports typically include information regarding its future performance, where investors are able to get a thorough understanding of the company's current position in its respective industry and the company's plans for future growth.As annual reports contain comprehensive information about the performance of the companies, and they are publicly available, we use the annual reports to conduct the research on financial crisis prediction.We extract structured financial data (hereinafter referred to as financial indicators) and unstructured financial text (hereinafter referred to as financial text) from the annual reports of listed companies in China for financial crisis prediction.We constructed a crisis prediction model based on a deep neural network.Compared with other work, our work has some unique characteristics.First, we combine the financial indicators and financial text for financial crisis prediction, while most existing methods rely only on the financial indicators and ignore the financial text.We will show that financial text is valuable in detecting the financial crisis.Second, we use deep learning based models to extract features from financial indicators and financial text for the prediction of the crisis.CNN and LSTM are two widely used deep learning models, each with its own advantages and disadvantages.CNN is mainly used to identify twodimensional graphics that are invariant to displacement, scaling, and other distortions, while LSTM is suitable for processing time series data.Mixing CNN and LSTM models can combine the advantages of both to form a more powerful model with a wider range of applications.The novelty of this hybrid model lies in its simultaneous utilization of the characteristics of two different types of models to achieve higher accuracy and efficiency.
Compared with the statistical methods and traditional machine learning methods, deep learning based models are more capable of extracting deep features and capturing the complex nonlinear relationships between data.Experimental results also show that our method outperforms existing methods.The major contributions of this work are as follows.First, we construct a financial indicators matrix, and use a convolution neural network to extract the hidden features, especially hidden financial ratios based on these ratios.Second, we use a deep learning method to encode the information in the financial text.Third, we combine the features extracted from financial indicators and text to predict financial crisis.Fourth, we conduct a set of comprehensive experiments to verify the effectiveness of the proposed method.Experimental results show that our method shows satisfactory performance.

RELATED WORK
There has been a long history of research on the topic of financial crisis prediction (FCP).According to the underlying data, existing methods can be divided into methods based on financial indicators, methods based on financial text, and methods combining financial indicators and financial text.

Methods Based on Financial Indicators
The research on financial risks in fields such as finance and accounting is generally based on financial data analysis, among which there are three mainstream methods for analyzing financial indicators, including factor analysis, comparative analysis, and ratio analysis.The ratio analysis method is an analytical method that calculates the ratios of different financial indicators, analyzing the financial changes of listed companies through changes in ratios.There are three major subcategories, including composition ratios, efficiency ratios, and related ratios.The comparative analysis method is to compare the same financial indicators of the same company in different periods, and analyze the operating conditions of listed companies during this period, in order to determine the impact of changes in financial indicators.Factor analysis method is a commonly used financial analysis method that determines the direct correlation between factors that affect financial indicators in order to infer the influencing factors related to the finance of listed companies [3][4][5][6].Early research on predicting corporate financial risks typically used individual financial indicators for analysis, but the financial situation of a company is influenced by various relevant factors.Altman introduced the Multivariate Discriminant Analysis (MDA) method to FCP for the first time, and proposed the ZETA model [7].Logit analysis was proposed by Martin [8] for the prediction of bank failures, and by Ohlson [9] for prediction of business failures.Dimitras et al [10] provided a thorough review of statistical methods used for prediction of business failures.Other methods include AHP [11].Logistic regression is widely used in credit scoring [12].A number of machine learning models had been introduced for problems related to FCP, such as bankruptcy prediction and business failure prediction.According to Vadlamudi [13], neural network (NN) was the most commonly used technique.NN is inspired by the biological networks of the human nervous system.As a nonlinear mathematical approach, NN often outperformed other single classifiers when testing complex data patterns as evidenced by [14].Other machine learning techniques include support vector machines (SVM) [15], support vector regression (SVR) [16], decision trees [17], genetic algorithms etc. [12].Deep Learning is a particular type of machine learning that consists of multiple NN layers.It provides high-level abstraction for data modelling.Lanbouri [18] used a hybrid DBN with SVM for financial distress prediction.Hosaka [19] used the CNN model to predict the bankruptcy risk by converting the financial indicators into images.

Methods Based on Financial Texts
Financial texts are usually studied using text mining techniques.Text mining is a new research field that has emerged in the field of computer science.The usual research approach is to extract the inherent information, special patterns of data, internal connections between language and text, text patterns, and development trends from massive, unstructured text that is carried by language and text, efficiently extracting valuable knowledge from language and text files, then applying this knowledge better.The text mining technology in the computer field has extremely practical significance for the efficient management and effective utilization of language, text, and text information, and has become a key means of text information management [20].Text mining is mainly used for research on company performance disclosure, audit reports, and listed company conference calls.Predictions based on text mining are just emerging to be investigated rigorously in the recent years [21][22].Antonina [23] used language analysis methods to study the information, style, language and readability of financial reports of leading companies in the telecommunications industry.Tetlock [24] uses Harvard dictionary to link the media content in the WSJ with the level of the stock market.Henry [25] constructed a bag of words about positive and negative tones and found that the tone of earnings announcements would arouse market reactions.Garcia [26] used LM's vocabulary list to measure the economic column of the New York Times from 1905 to 2005.In 2019, Rastin [27] used CNN and RNN models based on financial texts to predict whether the company has a default risk.

Methods Combining Financial Texts and Indicators
Combining multiple classifiers has been regarded as a new direction to develop effective FCP systems [12].Existing combination methods fused the results of multiple classifiers based on the financial indicators.Lanbouri [18] adopts a two-stage hybrid model that integrates deep learning and SVM for financial distress prediction.Huang [28] also studied the hybrid DBN-SVM model.In summary, there are currently quite a few studies on financial risk prediction, but there are few studies using financial texts for risk prediction, and there are even fewer studies combining financial data and financial texts.This article combines financial data with financial texts and uses deep learning methods to extract deep level features.
Compared to existing methods, our research utilizes more comprehensive data and methods that are more effective.

Model Metrics
In binary classification problems, there are usually model performance evaluation indicators such as model prediction accuracy, true positive rate, true negative rate, accuracy rate, F1 index [29].The most basic indicator is the accuracy of model prediction.Next, we will introduce the above indicators separately.
(1) Prediction accuracy.The definition of model prediction accuracy is as follows: given a training set D with a sample size of m, where each sample in the training set is x i and its corresponding label is y i , the prediction accuracy formula of model f is as follows.
I is a segmented function, I(1) =1 and I(0) =0.The prediction accuracy of the model represents the proportion of correctly classified samples to the total sample size.
(2) True positive rate, true negative rate, and accuracy rate.
True positive rate is also known as recall rate, while accuracy rate is also known as precision rate and precision.A binary classification task can combine all samples based on their true category and model classification results into four cases: true case, false positive case, true negative case, and false negative case.TP, FP, TN, and FN represent their corresponding sample numbers, and the confusion matrix shown in Tab. 1 can be provided.Based on these data, it can be defined as: true positive rate R = TP/(TP + FN), true negative rate K = TN/(TN + FP), accuracy (precision) P = TP/(TP + FP).The true positive rate represents the proportion of pairs in all positive cases, measuring the classifier's ability to recognize positive cases.Accuracy represents the proportion of true positive cases to the total number of predicted positive cases.The true negative rate represents the proportion of paired pairs in all counterexamples and measures the classifier's ability to recognize counterexamples.
(3) F1-score.We need to comprehensively consider various factors when evaluating the model, so the F1 value has emerged.Its definition is as follows.
F1 score is equal to the harmonic mean of precision and true positive rate.According to the nature of the harmonic mean in mathematics, the harmonic mean has a higher concern for the smaller value in the calculation factor, that is, when either of the model's precision and true positive rate is smaller, the F1 score of the overall model will become much smaller, although the value of the other party may be larger.This property precisely meets our needs for model evaluation, which requires the model to balance precision and true positive rate.Therefore, this paper uses F1 score as one of the main indicators for evaluating financial risk prediction models.At the same time, we also consider including the true negative rate as the main evaluation indicator, because in the experiment, not only positive cases but also negative cases need to be considered.

RESEARCH METHODOLOGY 3.1 Data Sources
We construct a dataset, called CFDC (Chinese Financial Data Corpus), to study the financial crisis encountered by corporates in China.We will use this dataset to train models to predict corporate financial crisis.In China, if the net profit of a listed firm is negative in two consecutive years, the firm will be labeled as special treatment (ST).According to the China Securities Supervision and Management Committee (CSSMC), negative net profit will increase the possibility of corporate failure.Hence, we follow the common practice [30] and treat ST as in financial crisis.The rest of the listed firms that have not been labeled as ST are regarded as non-crisis samples.We obtained a set of financial indicators from the Join Quant database [https://www.joinquant.com] about all listed companies in China from 2005 to 2019.These financial indicators can be divided into two parts: basic indicators and extra indicators.The basic indicators are mainly extracted from the financial statements of the annual reports, i.e. balance sheets, income statements, and cash flow statements.We get 214 basic indicators in total.Balance sheets is an accounting report that reflects all assets, liabilities, and owner's equity of a company over a certain period of time.It is a static manifestation of a company's operational situation.Income statement is an accounting statement that reflects the production and operating performance of a company over a certain period of time.The operating performance of a company over a certain period of time can be shown as profit or loss.Therefore, the income statement is also known as the income statement.The income statement comprehensively reveals the various income, expenses, costs or expenses realized by the enterprise over a certain period of time, as well as the profits or losses realized by the enterprise.Cash flow statement is a financial statement that reflects the impact of business activities, investment activities, and financing activities on a company's cash and cash equivalents during a certain period.This report shows how the balance sheet and income statement affect cash and cash equivalents, and analyzes them from the perspective of the company's operations, investments, and financing.In addition, we considered some other important financial indicators, such as total market value, year-on-year growth rate of operating income, year-on-year growth rate of net profit, earnings per share, etc.There are 36 extra financial indicators.In total, the number of financial indicators is 250.We obtain the annual reports of listed companies from 2000 to 2019 from Net Ease Finance [https://money.163.com].Following the common practice, we only use the chapters titled "Discussion and Analysis of Operation" in the annual reports.In total, there are 379 companies in crisis in the data set, with a total of 2791 samples, and 2781 normal companies with a total of 18917 samples.The total number of samples in the dataset is 21708.Each sample corresponds to an annual report of a company, composed of a set of indicators and financial text.Note that we use the data of a company at time t to predict the probability of the company being in crisis at t + 3. Given a company, we obtain the values of all indicators from the Joint Quan database.

Data Cleaning
We conduct data cleaning on the indicators and text as follows.
Indicator removal.Among all the samples, if an indicator has more than 30% missing values, the indicator will be discarded.After removal, 146 financial indicators are left.
Sample removal.Given a sample, if more than 50% of its indicators are null, the sample will be deleted.After this step, 17107 samples are left.
Filling missing values.For the remaining missing values, we fill missing values with the mean of the indicators, plus a random bias.
Indicator normalization.Values in each indicator are normalized to be in [0, 1] using the min-max normalization method.
Stop words removal.Punctuation, interjections, stop words and numbers in financial text are removed.
Field-specific terms removal.There are many terms unique to some specific industries, which offer little information for analysis of texts.If a term appears less than certain times in the corpus, the term would be removed.

Models 3.3.1 Processing Financial Indicators
In real-world financial analysis, financial ratios have proven to be very useful in analyzing the financial status of the companies.However, financial ratios are largely unexplored in existing work.In this work, we put much focus on the financial ratios.Practitioners in financial fields frequently use some financial ratios, such as current ratio, asset-liability ratio, etc.However, ratios used in real world practice are rather limited, there are probably some other ratios, and other features derived from ratios may help in analyzing financial status.In this section, we try to explore more features based on financial ratios using deep learning methods.Construction of the matrix of ratios.A financial ratio is a relative magnitude of two selected financial indicators taken from a company's financial statements.For example, the current ratio measures a company's ability to pay off short-term liabilities with current assets: / Currentratio Currentassets Currentliabilities  .In order to explore more ratios, we constructed a matrix of financial ratios, where each row and each column correspond to an indicator, respectively.In addition, we append a row and a column of constant indicator.Each cell is a ratio obtained by dividing the two financial indicators.Let n be the number of indicators, then the size of the ratio matrix is (n + 1) × (n + 1).In Tab. 2 we show an example of the constructed matrix of ratios.I 1 , …, I n are financial indicators.It can be seen that the matrix includes all possible ratios.What is more, it also includes the set of financial indicators and their reciprocals.We will use this matrix for further analysis.
Table 2 The matrix of financial ratios Feature extraction on the matrix of ratios.Although the matrix includes much more ratios than existing used ratios, we may still need to explore more features.For example, different ratios may be combined to generate new features.Considering that deep learning models are widely used in extracting high-level abstract features, we exploit CNN (Convolutional Neural Network), a powerful deep learning model to extract features on the matrix.CNN is widely used in computer vision and has become the state of the art for many visual applications such as image classification.Our choice of CNN model is inspired by the fact that, in computer vision, a CNN model accepts an image, which can be understood as a matrix with many layers, as input, and extracts features deep in the input data.In this work, we apply the CNN model on the matrix of ratios, and obtain features tailored to the training dataset.The process of the model is shown in Fig 1 .A number of CNN layers are applied on the financial matrix.Each layer is composed of a convolution step followed by a pooling step.After that, we use a fully connected neural network to generate a vector, which is taken as the features generated from the data.

Processing Financial Text
Text in financial documents has proved to be useful in financial prediction.In this work, we incorporate financial text to predict financial crisis.We use the word 2vec Skip-gram model [31] to get the embeddings of words in financial texts.We start with the public available word embeddings trained by Tencent [32], and then retrain the word embeddings using our financial text, and finally get the financial embeddings of the words.We use the bidirectional LSTM model to encode the tokens in the financial text, and global attention similar to Luong et al. [33] is applied, so that more attention will be paid to the important tokens.The context vectors obtained will be taken as encodings of the input text and fed into the model.

Processing Financial Text
In this section, we combine financial indicators with financial texts to build a financial crisis prediction model.
The overall model structure is shown in Fig. 2. The input of the model consists of two parts: the financial indicators, and the text of "Business Discussion and Analysis" section in annual report.The lower part of the figure uses an LSTM model to encode the text, and then imposes an attention mechanism.The upper part processes the matrix of ratios as discussed before, and a vector is obtained encoding the information of the matrix.The vectors obtained from the financial indicators and the text are then combined together to get the final vector.The vector is then sent to a simple MLP (Multi-layer Perceptron), which produces the prediction result.We tried two ways of combining financial data with financial texts.
(1) Concat: the vectors obtained from the financial text and that from financial indicators are concatenated.
(2) Addition: the weighted sum of the vectors obtained from the financial text and that from financial indicators is computed.
Through the experiment, it is found that the Addition method is better than the Concat method, so we use the Addition method in our experiments.

EXPERIMENT
In this section, we conduct comprehensive experiments on the CFDC dataset we constructed.We first use only the financial indicators and compare traditional machine learning methods and CNN based methods.Then, we incorporate financial text, and compare the combination model with other models.

Processing Financial Text
We first try traditional machine learning methods.The methods are XGBoost and SVM.We run several groups of experiments to determine the parameters of SVM and XGBoost models.In each group of experiments, we change the values of one parameter, and keep other parameters unchanged.We try two sets of financial indicators Ind1 and Ind2.Ind1 has only 47 indicators, and does not contain any indicators with missing values while Ind2 is the set of financial indicators obtained after data cleaning as described before.It has 146 indicators in total.The primary performance measures used in this work are accuracy (Acc, the overall rate of correctly classified instances), true positive rate (TPR, the rate of correctly classified positive instances), true negative rate (TNR, the rate of correctly classified negative instances), and F1-score (F1).The experimental results are shown in Tab. 3. From the table above, we can find that there is a big gap on the performance of the models using different sets of indicators.Using Ind1, we can achieve a high TPR but a low TNR, with both methods.In fact, both methods tend to classify all samples into the group without crisis.Using Ind2, we get a balance between the two classes.We think that TNR is more important in our problem, so we use Ind2 in the following experiments.When comparing SVM with XGBoost, we find that SVM has a better result on TNR, while XGBoost does better on TPR.

Results on Financial Text
The performance of neural networks is influenced by a series of parameters.In order to select the optimal parameters to fit the model, we set the range of parameter values.For example, the number of hidden layers k ∈ {1, 2, 3} in LSTM, the number of neurons n ∈ {64, 128, 256, 512, 1024} in hidden layers, the dimension L ∈ {100, 200, 300} in word vectors, and the learning rate λ ∈ {10  From the table, it can be seen that the F1 value increases significantly after adding attention mechanism, while the word vector dimension increases.The increase in F1 value is not significant.When taking different word lengths, the experimental results also vary.We adopt the approach of deleting more and increasing less for the word length of different financial texts.For example, taking a word length of 1300, assuming that when the financial text has 1289 words, the word length of the financial text is less than 1300, the remaining 11 words will be supplemented with 11 empty vectors.And words that exceed 1300 will be eliminated.Different word lengths have different effects.If the length of the text is long, the attention to useful information extracted may not be sufficient, while if the length is short, the extraction of important information may not be sufficient.Therefore, the length of the word is also considered one of the important parameters.For a more intuitive display, provide a graph of the changes in various evaluation indicators when the word length changes.The results are shown in Fig. 3.It can be seen from the figure that Acc, F1 and TNR increase slightly when the length increases from 800 to 1200.After that, TPR increases while TNR and F1 decrease rapidly when the length increases from 1200 to 1500.This is probably because much noise is introduced when the text expands.In the following experiments, we set the length of the financial text to be 1200.The number of dimensions of word vector (#dim) is an important parameter.We empirically choose two numbers of dimensions: 100 and 200, and report the results below in Tab. 5. We can see from the table that the performance is better when #dim = 200.Therefore, we set the #dim to be 200 in the following experiments.We also evaluate whether the attention mechanism has impact on the performance of the model.As shown in Tab.6, with attention mechanism, there is an observable increase on Acc, TPR and F1.Thus, conclusions can be made that attention mechanism improves the performance.

Combining Financial Indicators and Texts
In this section, we combine financial indicators and financial texts for financial crisis prediction, and compare the results of the combination method with other methods.These models are: Ind-SVM: this method is based on financial indicators, using SVM as classifier.Ind-XGB: this method is similar to Ind-SVM, but uses XGBoost as classifier.Ind-CNN: this method first builds a matrix of ratios on financial indicators, and then uses a CNN model to extract features, which follows two FC layers and a soft max layer as classifier.Txt-LSTM Att: the method is based on financial text, using LSTM and attention mechanism.
Ind + Txt: Financial indicators and financial text are combined in the second way.Parameters of the CNN model are tuned according to the performance.The final parameters are as follows: the size of the convolution kernel is 4OE4, 5OE5 and 6OE6, the size of the pooling layer is4OE4, the number of convolution kernels is 300, the number of layers is 4. The sizes of the FC layers are 64 and 128, respectively.The parameters of the LSTM model are kept the same as described in Section 4.2.
The results are shown in Tab. 7. From this set of experiments, we can see that the Ind-CNN model outperforms the Ind-SVM and Ind-XGB models, which is probably because CNN model can better extract more useful features from the data.Generally, the financial text-based Txt-LSTM-Att model outperforms the models based on the financial indicators, indicating that texts from financial report contain valuable information for financial crisis prediction.Especially, the Txt-LSTM-Att has the best results on TPR.It can be seen from the table that the combination of financial indicators and financial text exhibits more potential than financial indicators alone, or financial text only.It has a high TNR and high TPR, and the best F1 score.This is probably because the Ind + Txt model combines Ind-CNN and Txt-LSTM-Att, therefore, it has the merits of both Ind-CNN and Txt-LSTM-Att.

CONCLUSION
This article studies a financial risk prediction model based on financial data and financial texts.Three models have been proposed.The first model uses financial data to model using CNN, SVM, and XG Boost, respectively.The second model uses financial text to model using LSTM + attention mechanism.The third model combines financial data with financial text to model.We attempted to use two network structures for the third model, one consisting of CNN convolutional neural network and LSTM short-term memory neural network.Another network structure consists of a single hidden layer feedforward neural network and an LSTM short-term memory neural network.The experimental results showed that compared to the model that only used financial text for financial risk prediction, the combined model with financial data matrix increased F1 by nearly 5 percentage points, while the combined model using financial data vector only increased F1 by about 1 percentage point.In both cases, the improvement in model performance is statistically significant at the 1% level.In models that did not use financial text, the average increase was 13% after using financial text.Statistical testing also shows that the financial data matrix contains much more information than the financial data vector, and all useful information contained in the financial data vector is also included in the financial data matrix.These findings suggest that further analysis should only focus on the financial indicator matrix.The limitations of learning mainly include the following two points: (1) This article only considers the relevant information of a single chapter in the financial report, and cannot mine all the financial report text information in the entire financial report.It is necessary to consider different chapter combinations for modeling from the perspective of combining financial report chapters.
(2) This article only considers one issue of the annual report, and the historical sequence of the annual report can also be included in the model.It is possible to extract more and better information that can provide evidence for financial risks from the sequence, thereby improving model performance.

Figure 1
Figure 1 CNN model based on financial indicator matrix

Figure 3
Figure 3 Performance when the length of text changes

Table 1
Confusion matrix True value

Table 3
Experimental results on SVM and XGBoost

Table 4
Partial experimental results

Table 5
Experimental results with different dimensions

Table 6
Experimental results with or without attention

Table 7
Prediction results of various models