Maritime Data Mining for Marine Safety Based on Deep Learning: Southern Vietnam Case Study

2.1. Overview of maritime data mining connected to classification and analysis algorithms / Pregled rudarenja podataka u pomorstvu povezanih s algoritmima klasifikacije i analize

Figure 1 Maritime traffic state prediction system

Slika 1. Sustav za predviđanje stanja pomorskog prometa

In maritime big data management, marine data is categorised into one or more given classes using classification models. These models are trained on a historical dataset with labels [6]. Assigning labels to data items is the process of classification. The goal of the classification challenge is to identify a variety model that allows the determination of the class to which the latest information belongs. In this section, we examine some modern analysis and application methods for processing maritime big data [6-19] as follows:

Remark 1: Maritime data is collected from many different sources and does not have integrated links. Therefore, it is necessary to develop an integrated management system to improve the efficiency of data exploitation to support maritime safety, such as predicting the possibility of collisions and monitoring vessel mooring wharves.

2.2. Specific time-series data based AIS / Specifični AIS temeljen na vremenskim serijama podataka

In particular, the vessel kinematic information, including latitude (lat), longitude (lon), speed over ground (SOG), and course over ground (COG), plays a critical role in evaluating optimal navigation routes, and predicting the future path of a vessel over specific time-series data based on relevant historical data requires analyzing an array of AIS data [13]. It is denoted by equations (1), (2), (3) [14-15]

[TeX:] X_{t} \triangleq \left\lbrack \text{lat},\text{lon},\text{SOG},\text{COG} \right\rbrack^{T}$X_t\triangleq {[\mathrm{\text{lat}},\mathrm{\text{lon}},\mathrm{\text{SOG}},\mathrm{\text{COG}}]}^T$(1)

The vessel's historical path (The original AIS data for the MMSI vessel, code of 525100764, expressed by Southern Vietnam Maritime Safety Corporation), which is expressed in Table 1, is represented by a sequence of observation points [TeX:] \{ X_{t_{0}},X_{t_{1}},\ldots,X_{t_{T}}\ \}$\{X_{t_0},X_{t_1},\dots ,X_{t_T}\}$, where [TeX:] t_{i}$t_i$ < [TeX:] t_{j}$t_j$ if [TeX:] i\ < \ j$i<j$. Therefore, it is necessary to carry equally sampled observed data to obtain a series of [TeX:] T + 1$T+1$ as follows:

[TeX:] X_{0:T} \triangleq \{ X_{t_{0}},X_{t_{0} + \mathrm{\Delta}t},X_{t_{0} + 2*\mathrm{\Delta}t}\ldots X_{t_{0} + T + \mathrm{\Delta}t}\}$X_{0:T}\triangleq \{X_{t_0},X_{t_0+\Delta t},X_{t_0+2*\Delta t}\dots X_{t_0+T+\Delta t}\}$(2)

The process of encoding complicated vessel motion data in this space feature poses a significant challenge. Therefore, the solution used is to expand the feature space by one higher dimension. The "four-hot" representation vector [TeX:] h_{t}$h_t$ is used to separate lat, lon, SOG, and COG data into [TeX:] N_{\text{lat}},N_{\text{lon}},N_{\text{SOG}}$N_{\mathrm{\text{lat}}},N_{\mathrm{\text{lon}}},N_{\mathrm{\text{SOG}}}$ and [TeX:] N_{\text{COG}}$N_{\mathrm{\text{COG}}}$ bins [16], respectively. The vector [TeX:] h_{t}$h_t$ is expressed by

[TeX:] h_{t} \triangleq \left\lbrack 1_{t}^{\text{lat}},1_{t}^{\text{lon}},1_{t}^{\text{SOG}},1_{t}^{\text{COG}} \right\rbrack^{T}$h_t\triangleq {[1_t^{\mathrm{\text{lat}}},1_t^{\mathrm{\text{lon}}},1_t^{\mathrm{\text{SOG}}},1_t^{\mathrm{\text{COG}}}]}^T$(3)

Table 1 The original AIS data sample for the MMSI vessel code of 525100764

Tablica 1. Izvorni uzorak AIS podataka za MMSI brodski kod 525100764

Remark 2: Depending on the weather and traffic, different features of vessels traveling along the comparable route will be observed. In case of dealing with large inertia vessels, and complex propulsion systems, it is necessary to predict the safe routes.

3.1. Dynamic visualization of the vessel movement tracks in Vungtau port / Dinamička vizualizacija putanja kretanja plovila u luci Vungtau

Nowadays, vessels are becoming complex sensors concentrated as the maritime industry's digital revolution gathers increasing volume to support more energy-efficient marine and vessel operations and support handling the challenges of new legislation. The result of the combination of modern communication systems and advanced sensor technologies is significantly improved vessel connectivity, which allows for the collection and analysis of a large amount of operational data. Specifically, the synchronization and analysis of data from various sources will undoubtedly speed up decision-making for operators and improve vessel performance management in critical areas including energy and fuel management, emissions control, machinery and equipment monitoring [17], and route optimization. Thus, data mining will benefit the shipping sector by providing fresh insights and added value to support improved decision-making, asset tracking, predicting, and fleet-wide optimal application. The methodology used in this study focused on mining maritime traffic from historical vessel data, and consists of two stages: data collection and classification in the first stage, together with the required measurement metrics; analysis, and prediction of marine traffic states using tools or algorithms in the second stage.

Figure 2 Dynamic visualization of the vessel movement tracks in Vungtau port

Slika 2. Dinamička vizualizacija putanja kretanja plovila u luci Vungtau

Maritime traffic-related data is collected from various sources from existing fixed monitoring systems. As we know, static data, dynamic data, and auxiliary data are three types of data, based on the sample of data collection types. One of the sample data used in this paper is described in Table 2. For clarity, we used sample data of dense maps visualized for July 27, 2019, and Fig. 2 shows the dynamic visualization of the vessel movement in Vietnam's southern region. Obviously, the more data is collected, the greater the chances that the system will estimate traffic conditions timely and accurately. To be more precise, we use the vessel's dynamic visualization to estimate traffic circumstances almost in real-time. This allows us to provide suitable models for managing and predicting maritime traffic conditions, even when data segments are missing.

Table 2 Classification of data collection

Tablica 2. Klasifikacija prikupljanja podataka

Visualizing the initially collected AIS data helps the authors take an overview of the collected dataset. Consequently, the data preprocessing avoids missing data, which leads to the loss of the crucial features of the dataset. The authors determine the coordinate area in order to extract suitable data for the evaluation process based on the data visualization. In addition, data fields (such as ship name, call sign, and band) that do not affect the goals of training the prediction model are removed in order to increase processing speed. In this study, the visualization data array focuses on vessels with continuous paths and docking in the coastal area of Vung Tau City, with features of type and MMSI identifier and vessel dynamic data (lat, lon, SOG, and COG).

The labeled data, which expressed the Fairway Maritime Traffic (FM-Traffic), may be used for prediction models in data mining techniques. The model assesses the FM-Traffic and channel conditions where time-series data is not available due to previous vessel data. Fig. 2 depicts the suggested structure for the marine traffic state prediction model, which is summerized as follows:

Upon obtaining the above-described generic structure (Fig. 1), we need to establish the most important data to efficiently train the model. The fact that the traffic in each channel differs and changes regularly is one of the problems here. For the spatial path, we divide the route network into channels based on the ENC, where each channel is short enough to take into account that the variance of traffic conditions at any location in a channel can be ignored. In terms of time, we divide the time into time frames based on which the collected data is integrated and analyzed. Following the data separation method mentioned above, this model is quite weightless and yet possible in practice. This basic format is simple enough that it can be collected and integrated with any device that uses VHF frequencies, as presented in our previous work [25].

3.2. Management integration system / Integracijski sustav upravljanja

The management system consists of four main components, namely the API server, the application, the computing server, and the database server, as shown in Fig. 3 below:

Figure 3 Structure of the management system

Slika 3. Struktura sustava upravljanja

API server: The core component of the system is the server, which processes user requests. The server is established in NodeJS – an open-source, cross-platform framework. When users request the application, the data is retrieved from the database, processed, and then sent back to the users. The request life cycle includes receiving and identifying user requests, validating data, and processing requests.

Computing server: The computing server processes the data submitted by users or the AIS. Hence, the computing server performs calculations to return the speed corresponding to the vessel’s navigation status on the application map. At the end of each cycle, the Computing Server will update the user's speed and reputation score. If for some routes there is no enough data to calculate the speed, the computing server will refer to other data sources from resource APIs to ensure that the vessel's navigational status is always fully displayed. Moreover, the directory structure of the computing server is almost identical to the directory structure of the API Server. The computing server only computes and stores data, while the API server defines the endpoint access points.

Database server: The database server has the task to store data from the AIS identifying information system and process and compute data on the system. As this is a real-time system with big data, the database must have the following features: easy access to vast information; the need for a geospatial database for the navigation system; and confidentiality to protect the vessel’s data.

Application: This application is expressed on an online platform that handles communication with users. The program allows users to view the status of vessels. Additionally, the application is responsible for determining the vessel's route and collecting data for communication with the API server.

The system is architecturally constructed in a modular form with great flexibility, which enables scale growth with sensor stations (AIS, HMIS, etc.) and working positions (operation desk, training desk). At the same time, the system enables information-sharing interfaces with traffic management centers. Moreover, the system provides features suitable for each task defined based on the primary purpose of the user, including AIS subsystem, VHF, MIS, ENC, and Hydrometeorological data [24-25]. In conclusion, the general description of the system components demonstrates the relationship between the system utilization and each function, but in this study we focus on improving the application to support maritime safety.

3.3. Maritime traffic state prediction model based on LSTM network / Model predviđanja stanja pomorskog prometa temeljen na LSTM mreži

In the past ten years, concerns about maritime traffic safety and security have become evident due to the difficulties created by the increasing demand for additional vessels with greater capacity and velocity. To ensure the navigational safety, prevent collisions, and improve the effectiveness of vessel management, predicting the trajectory of vessels is essential. A relatively recent development for complex geographic applications is the addition of effective machine learning technologies to accurately predict trajectories. However, the complexity of the maritime environment and issues with data quality, particularly in the Vietnam Sea Port, which has a high density of vessels, hinder the reliable vessel trajectory predictions. On the other hand, with the system structure selected in Figure 3, the input data is processed and analyzed numerically, stored in Resource APIs, and then aggregated and filtered, with the data fields being separated. Subsequently, the 04 data fields (lat, lon, SOG, COG) selected for the prediction model are also the input data of the Maritime traffic state prediction model [25]. In addition, the MTSP model has not yet been implemented in any maritime management system in Vietnam, which motivates us to propose a solution based on the support of Deep Learning to provide the system with the following superior features. In this study, we suggest the MTSP model based on the LSTM network (LSTM is one of the deep learning algorithms as mentioned) to evaluate suitable paths for vessels along routes and experiments established on data collected from the AIS system through Resource APIs provided by Southern Vietnam Maritime Safety Corporation. The proposed model used for analyzing maritime traffic data has the following characteristics:

The LSTM network sequentially computes the input vessel path data string [TeX:] X_{l}$X_l$ with the hidden vector [TeX:] Η_{\mathcal{l}} \triangleq {\{ h_{t}\}}_{t = 1}^{\mathcal{l}}${{\rm H}}_{𝓁}\triangleq {\{h_t\}}_{t=1}^{𝓁}$, in which the memory cell corresponding to the input vector (at the current time step [TeX:] x_{t}$x_t$) and the hidden state (at the previous time step [TeX:] x_{t - 1}$x_{t-1}$) update the hidden state inside [TeX:] h_{t}$h_t$ expressed by [20]

[TeX:] \begin{matrix} i_{t} = \sigma\left( U_{i}x_{t} + W_{i}h_{t - 1} + b_{i} \right) \\ f_{t} = \sigma\left( U_{f}x_{t} + W_{f}h_{t - 1} + b_{f} \right) \\ o_{t} = \sigma\left( U_{o}x_{t} + W_{o}h_{t - 1} + b_{o} \right) \\ \end{matrix}$\begin{array}{c}\hfill i_t=\sigma (U_i{x}_t+W_i{h}_{t-1}+b_i)\hfill \\ \hfill f_t=\sigma (U_f{x}_t+W_f{h}_{t-1}+b_f)\hfill \\ \hfill o_t=\sigma (U_o{x}_t+W_o{h}_{t-1}+b_o)\hfill \end{array}$(4)

where [TeX:] \odot$\odot$ represents the element product, [TeX:] \sigma$\sigma$ describes the sigmoid function, and [TeX:] \tanh$tanh$ is the hyperbolic tangent function. Besides, [TeX:] i,\ f$i,f$ and [TeX:] o$o$ indicate the input gate, forget gate, and output gate, respectively. We get, [TeX:] {\widetilde{c}}_{t}${\tilde{c}}_t$ and [TeX:] c_{t}\text{\ ϵ\ }\mathbb{R}^{q}$c_t\mathrm{\text{ϵ}}{ℝ}^q$ express the cell input activation vector and cell state, defined as follows:

[TeX:] \begin{matrix} {\widetilde{c}}_{t} = tanh\left( U_{c}x_{t} + W_{c}h_{t - 1} + b_{c} \right) \\ c_{t} = f_{t} \odot c_{t - 1} + i_{t} \odot {\widetilde{c}}_{t}\text{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ } \\ h_{t} = o_{t} \odot tanh\left( c_{t} \right),\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \\ \end{matrix}$\begin{array}{c}\hfill {\tilde{c}}_t=tanh(U_c{x}_t+W_c{h}_{t-1}+b_c)\hfill \\ \hfill c_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t\hfill \\ \hfill h_t=o_t\odot tanh\left(c_t\right),\hfill \end{array}$(5)

The input weight matrices are represented by [TeX:] W_{s}(W_{i},W_{f},W_{o},U_{c})\ $W_s(W_i,W_f,W_o,U_c)$and [TeX:] U_{s}(W_{i},W_{f},W_{o},W_{c})$U_s(W_i,W_f,W_o,W_c)$, with [TeX:] b_{s}(b_{i},b_{f},b_{o},b_{c})$b_s(b_i,b_f,b_o,b_c)$ being the bias terms. The weight matrix subscript indicates the input-output connection. [TeX:] W_{f}$W_f$ is the implicit forgetting gate matrix, and [TeX:] U_{f}$U_f$ is the input-forgetting matrix. The encoder codes the vessel's kinematic state sequence [TeX:] X_{\mathcal{l}}$X_{𝓁}$ one state at a time into a hidden state sequence. We employ an encoder-decoder architecture to solve the prediction problem of mapping one data sequence to another, specifically defining the mapping function [TeX:] F_{\mathcal{l,}h}$F_{𝓁,h}$. The initial encoding function [TeX:] E$E$ is represented by [20]

[TeX:] H_{\mathcal{l}} = E\left( X_{\mathcal{l}};\theta_{E} \right)$H_{𝓁}=E(X_{𝓁};{\theta }_E)$(6)

where [TeX:] H_{\mathcal{l}}$H_{𝓁}$ is the neural network parametrized by [TeX:] \theta_{E}${\theta }_E$ that maps input sequence [TeX:] X_{\mathcal{l}}$X_{𝓁}$ to an internal representation data sequence [TeX:] H_{\mathcal{l}} = \left\{ h_{t} \right\}_{t = 1}^{\mathcal{l}}$H_{𝓁}={\left\{h_t\right\}}_{t=1}^{𝓁}$. Each hidden state [TeX:] h_{t} \in R^{2q}$h_t\in R^{2q}$ combines bidirectional recurrent neural network (RNN) with a state of size [TeX:] q$q$.

The encoder layer computes the [TeX:] H_{\mathcal{l}}$H_{𝓁}$ representation of the input sequence, which created the context representation by the aggregation function. The decoder repeatedly uses this context representation to generate the output prediction. We use the average pooling over time (AVG) to reduce the sequence [TeX:] H_{\mathcal{l}}$H_{𝓁}$ to a single context vector as

[TeX:] z = col\left( z_{r} \right) \in \mathbb{R}^{2q}$z=col\left(z_r\right)\in {ℝ}^{2q}$, [TeX:] r = 1,\ldots,2q$r=1,\dots ,2q$(7)

for computing the mean value of each hidden unit. Each context feature [TeX:] z_{r}$z_r$ is defined as

[TeX:] z_{r} = \frac{1}{\mathcal{l}}\sum_{t = 1}^{\mathcal{l}}{{{(H}_{\mathcal{l}})}_{r,t},}\ t\epsilon\{ 1,\ldots,\mathcal{l\}}$z_r=\frac{1}{𝓁}{\sum }_{t=1}^{𝓁}{{(H}_{𝓁})}_{r,t},tϵ\{1,\dots ,𝓁\}$(8)

The symbol [TeX:] \theta_{D}${\theta }_D$ represents the parameterization of the autoregressive decoder function [TeX:] D$D$ to predict the future vessel path [TeX:] {\widehat{y}}_{j}${\hat{y}}_j$ at each period [TeX:] j$j$ with the previous state [TeX:] {\widehat{y}}_{j - 1}${\hat{y}}_{j-1}$ as follows [21]:

[TeX:] {\widehat{y}}_{j} = D\left( {\widehat{y}}_{j - 1},u_{j},z_{j},\psi,\theta_{D} \right)${\hat{y}}_j=D({\hat{y}}_{j-1},u_j,z_j,\psi ,{\theta }_D)$(9)

where [TeX:] u_{j}$u_j$ denotes the RNN hidden state with [TeX:] \psi$\psi$ being the planning descriptor and [TeX:] z_{j}$z_j$ being the context vector. Finally, the output prediction response [TeX:] {\widehat{Y}}_{h}${\hat{Y}}_h$ of length [TeX:] h$h$ is given by

[TeX:] {\widehat{Y}}_{h} = F_{\mathcal{l,}h}\left( X_{\mathcal{l}},\ \psi \right),${\hat{Y}}_h=F_{𝓁,h}(X_{𝓁},\psi ),$ (10)

To evaluate the quality of the prediction model response, the authors employ the root-mean-square-error (RMSE) [22-23] method to estimate the average error value of the squares between the predicted path [TeX:] \widehat{Y_{h}}$\widehat{Y_h}$ and the actual path [TeX:] Y_{i}$Y_i$, which is defined as

[TeX:] RMSE = \sqrt{\frac{1}{n}\sum_{i = 1}^{n}{(Y_{i} - \widehat{Y_{h}})^{2}}}$RMSE=\sqrt{\frac{1}{n}{\sum }_{i=1}^n(Y_i-\widehat{Y_h}{)}^2}$ (11)

LSTMs initially tried to replicate human decision-making by utilizing machines to process large quantities of data. Advanced LSTM systems introduce autonomous vessels, which can operate independently without human intervention and have a lower mistake rate than human-operated vessels. Deep learning is gradually altering the maritime industry's traditional operational processes, especially in mining maritime traffic from vessel data as mentioned in this paper.

4.1. LSTM network / LSTM mreža

Figure 4 MTSP model based on LSTM network

Slika 4. MTSP model temeljen na LSTM mreži

In this work, we use an encoder-decoder architecture to implement an input-output mapping function to predict future pathways. Based on a sequence of monitored states and past data describing the relative path of the vessel, the LSTM neural network architecture consists of three main phases: encoder, aggregation function, and decoder illustrated in the vessel path prediction in Fig.4. Therefore, to predict the safe path of a vessel in the selected case study, we built the proposed MTSP model based on the LSTM network, which includes the following steps:

This study employed the LSTM network to develop the path prediction model, executed in Python 3.6. The activity results are shown in Fig. 5, the setting up for 50 epochs training using a learning rate of 0.0005.

Figure 5 The regression and loss of LSTM network training process

Slika 5. Regresija i gubitak procesa testiranja LSTM mreže

4.2. Case study experimental results / Eksperimentalni rezultati studije slučaja

The optimal values including regression value attained 0.001075 in training, and the loss value reaches 0.0000039103. Besides, a set of historical vessel data from the AIS system is used as input data for the training prediction model to provide a safety path, as shown in Fig. 6. Thus, Fig. 6a illustrates the route taken by a vessel while departing from the wharf and moving towards the sea with the MMSI code 636018224 (indicated by the light-red line). Similarly, the blue line in the figure shows the path followed by the vessel while arriving from the sea to the wharf with the MMSI code 574999621. Finally, the dataset for the same type of vessel visualizing the maritime traffic conditions in VungTau port is indicated in Fig. 6b with vessels Type-1 (on the left side) and Type-3 (on the right side). Finally, the case study experimental results are shown in detail in Fig. 7, the predicted path of the vessel (red line) tracking follows the historical path (blue line) of this vessel.

a) The visualization samples of the historical vessel path visited (blue line) – exit (red line) from Vungtau Port ( based on AIS Data in 2019) / Uzorci vizualizacije povijesne putanje plovila (plava linija) – izlaz (crvena linija) luke Vungtau (na temelju AIS podataka 2019.)

b) The visualizing sample sets of historical AIS data from vessels / Vizualizacija skupova uzoraka povijesnih AIS podataka s plovila

Figure 6 Input data for the creation of a model to predict the path of a vessel in Vietnam's southern sea region

Slika 6. Ulazni podaci za izradu modela za predviđanje putanje plovila u južnom morskom području Vijetnama

The proposed model was tested for the cargo vessel traveling channel coming at VIETSOVPETRO wharf, where there is a complicated traffic situation with many vessel types. Based on historical data (blue line) of the same type moving along the channel, the LSTM prediction model extracts a safe path (red line) for the vessel to move in case of docking at the wharf. The case study experimental results are shown in detail in Fig. 7.

Figure 7 Predicted vessel path for arrival at VIETSOVPETRO wharfs in southern Vietnam

Slika 7. Predviđena putanja plovila za dolazak na VIETSOVPETRO pristanište u južnom Vijetnamu

4.3. Evaluations / Procjene

In general, the system displays marine traffic data calculation and updating and indicates adequate reaction times with initial operation at Vungtau Port, Vietnam. Tests were carried out on using the proposed model from the collected and distributed traffic data as well as obtaining traffic information from the AIS system. The results (Figure 6 and Figure 7) outline the management integration system, which, once implemented, will enhance the operational efficiency of the region's specialist maritime management. The system serves the common benefit of the community and guarantees national security and defense in terms of financial efficiency. Therefore, determining its effectiveness is difficult. However, from a socioeconomic standpoint, the initiative has the following consequences: 

In the future, the model can develop new application features in ensuring maritime safety by predicting the possibility of collision, predicting the risk of running aground, determining the closest point of approach, monitoring cargo anchorage locations, monitoring and indicating current vessel status to reduce risks to vessel, property, and people, as well as environmental pollution hazards. The advanced and modern technologies in state management methods can be used to increase the attractiveness and competitiveness of the seaport system. In addition, it actively contributes to the gradual perfection of specialized management in the maritime sector through international conventions to which Vietnam is a signatory. To this end, the concerns highlighted in Remark 1 have been addressed and the issues thus resolved.