C OMBINED AI C APABILITIES FOR E NHANCING M ARITIME S AFETY IN A C OMMON I NFORMATION S HARING E NVIRONMENT

The complexity of maritime traffic operations indicates an unprecedented necessity for joint introduction and exploitation of artificial intelligence (AI) technologies, that take advantage of the vast amount of vessels’ data, offered by disparate surveillance systems to face challenges at sea. This paper reviews the recent Big Data and


Introduction
Nowadays, maritime safety agencies are faced with many challenges varying from the high intensity of maritime traffic, vessel collisions in coastal areas, environmental risks from ships accidents, irregular maritime border-crossing and illicit activities at sea.Maintaining the required strategic and tactical level of maritime safety in a complex environment calls for support of sophisticated and smart ICT technologies, ready to assist in performing the operations of vessel traffic services and national rescue coordination centers (VTS/NRCC).The ever-increasing large amount of vessel data, collected through heterogeneous sensors and information sources demands appropriate structuring for exchanging them among collaborative agencies for undertaking joint operations and safety/security missions at sea and border.Therefore, in this paper we analyse the most important objectives that maritime safety sector strives to: 1. achieving greater maritime situational awareness through institutional networking among relevant agencies for Common Operational Picture at sea, 2. full exploitation of the latest innovative achievements, automated ICT technologies and big data science, capitalizing on versatile applications of AI for maritime purposes, such as anomalies detection and navigation predictions.
The goal of the paper is to present a case study EFFECTOR about maritime safety and two specific solutions conmbined AI features and how these need to be adapted for maritime context.Methological approach of this research reviews CISE as maritime safety EU initiative, the Big Data collected from various maritime sensors and shared among CISE network, with combined AI capabilities for the purpose of efficient response of maritime operative systems.Consequently, the paper unfolds as follows: Chapter 2 elaborates CISE in more details, while Chapter 3 reviews Big Data impacts on development of AI technologies in maritime environments.In Chapter 4 the case study presents EU project EFFECTOR with its specific solutions based on combined AI features for data/information fusion and vessel collision prevention.

CISE EU initiative in maritime safety
Considering that maritime safety critically relies on vessel surveillance systems and fast information flows networked via maritime authorities' national competent systems, the need for regional and international cooperation of European stakeholders has led to the establishment of the concept of Common Information Sharing Environment (CISE).The idea of establishing the CISE concept stems from the EUCISE2020, a test-bed project that triggers a creation of a common network for sharing and exchanging relevant maritime data and information between collaborating authorities.This concept was developed and extended through further innovation action projects supported by European Commission (EC) and aimed to improve the current performance in information sharing.Accordingly, the structure of the maritime CISE Data & Service Model defines in its vocabulary CISE Core and Auxiliary Entities concerning agents (person or organization), objects (vessel, operational asset), event (action, anomaly, incident), location, period, risks, documents (metadata), using XSD (XML Schema Definition) or UML (Unified Modelling Language).Being enhanced, the CISE Model introduces tasks, mission, operations, movement, maritime anomalies and sensors (AIS, radar, camera).For instance, the maritime risk type identifies crisis, border crossings, areas, vessels collisions, military and environmental risks (Mihailović et al., 2021a and2021b).In general, AI technologies, with their cognitive, forecasting and reasoning functions are intensively developed toward providing greater software support to human operators and agencies, increasing the level of automation in the maritime transport sector.The aim is to strengthen the maritime safety domain by utilization of prospective applications able to manage Big Data as: vessel route/paths control and optimization, vessel traffic surveillance, prevention of collision, possible fault/failure detection in ship operations, etc. Primarily, maritime AI applications retrieve the vast amount of data from different data source types, such as: fixed surveillance radar stations, patrolling and rescue ships, and most significantly, from electronic tracking system with automatic identification (AIS) for vessels movement and remote sensing systems.Furthermore, these Big Data, processed under Machine Learning (ML) or more specific Deep Learning (DL) approaches/techniques with optimization modeling enables the VTS/MRCC operator to increase the control efficiency on tactical level actions and assess the risks/accidents impacts at sea.Such processed data enable the highly value information for sharing within CISE Network.While regarding the vessel route data (such as AIS), ML is one of the research trends for anomalies detection.In the following subchapters are taken in consideration the works related on both of these research areas.

Maritime Big Data applications for vessels detection
The AIS is a cooperative information system that provides identification and position of ships in real-time, but its coverage is limited by the structure of the system itself.The most effective solution to cover the remote ocean areas are spacebased sensors, such as SAT-AIS (Helleren et al., 2012).The AIS and SAT-AIS are the most used tracking systems in Maritime Surveillance, which have proven to help and support the resolution of many problems in this area, but even with global coverage, the AIS has its downside caused by the monitoring limitation of only reporting vessels.Thus, AIS should be integrated by other vessel tracking data sources.One of the shared Big Data sources alterative to AIS is provided by satellites remote sensing, such as earth observation satellites and Synthetic Aperture Radar (SAR).These images cover all the globe and contain also ships that do not share AIS information.But unlike AIS, information in optical images is not explicit, and a specific process is needed to be done to detect the vessel in the images.The vessel information extraction from satellite imageries is driven by 3 main processes: object or vessel recognition (finds a vessel in the image), vessel classification (the class of the vessel) and vessel identification (Kanjir et al., 2018).The vessel recognition is the first step to extract vessel information from the images, it can exploit different types of algorithms, among these there is DL (Wang et al., 2018).
Even if in the past the image processing statistical techniques were more widespread, today it seems that the use of Neural Networks (NN) is gaining ground (Bentes et al., 2017), and in many works it is claimed that the latter provides advantages in terms of performance, and compared to statistics or even computer vision (Kanjir et al., 2018).In the next step, the classification of the vessels in almost all recent works converge in the use of AI algorithms.Most of these classifiers seem to use Support Vector Machines (SVM), and in recent years the trend is also in favor of using NN here.Instead, other works focus on Bayesian networks and other statistics and AI 150 35 TH BLED ECONFERENCE DIGITAL RESTRUCTURING AND HUMAN (RE)ACTION algorithms (Soldi et al., 2021).The information extracted using satellite images can be more effective in combination to those collaborative systems such as AIS, which include identification and higher temporal resolution (Achiri et al., 2018).Based on this information, it is possible to identify those vessels that omit the sending of AIS data, or that falsify them.In order to develop these operations, there exist different fusion techniques that have been studied (Fischer et al., 2010).An important point of these technique is the usage of the interpolation on the AIS data, used to estimate the AIS position at the moment in which the vessel is extracted from the satellite image (Nguyen et al., 2015).

Big data and AI solutions for maritime surveillance
In the previous chapter a series of information extraction techniques have been described, in this section the state of the art of anomaly detection algorithms are considered, grouping them by type of algorithms.SVM is one of the simpler machines learning methods, as it uses a separating hyperplane or a decision plane to demarcate decision boundaries among a set of data points classified with different labels.(Handayani et al. 2010) use SVM with Automated Identification System (AIS) from Port Kelang vessel, tracked for 3 months period and involving 367 tracks across 7 unique MMSI.By using these data, the paper assesses an accuracy of 90% of its techniques.Also, in (De Vries et al. 2012) SVMs is applied for detecting the outlying trajectories.The anomalies detection also takes advantage of Clustering, which is often used to extract patterns from the route and identify waypoints and classic routes.These routes, then, are used to describe the behavior of the vessels and to store this information in a sematic graph, that can be queried to find anomaly behavior as in (Varlamis el al. 2019).
Also (Dahlbom et al. 2007) explores trajectory clustering as a mean for representing the normal behavior of vessels.The approach uses spline-based clustering to overcome some issues in classical clustering.This approach breaks down the map into small zones where behavior patterns are detected.The most recent work that applies a similar approach is (Zhen et al. 2017) which executes a trajectory clustering, and then applies a Naïve Bayes classifier to detect anomalous vessel behavior.(Liu et al. 2015) separates the normal routes from AIS historical data and then extracts, using clustering, the normal trajectories and normal behaviors from that one with which the new data can be compared.The algorithms that have had the greatest growth and development in recent years are certainly those concerning NN in all variants, including also those that are defined as DL, which represents specific ML model with multiple layers of non-linear processing units, referred to systems with numerous serially connected layers of parallel connected neurons.
In (Nguyen et al. 2018) trajectory reconstruction, the anomaly detection and vessel type identification are the tasks by which the deep framework proposed in the work is demonstrated to be applied with effectiveness.The algorithm uses a Recurrent Neural Network (RNN) with latent variables showing that this algorithm is particular suited for time series processing.The RNN is also used in (Zhao et al. 2019) which adopts a hybrid approach using also clustering DBSCAN algorithm to extract the traffic patterns and trains the RNN composed of Long Short-Term Memory (LSTM) units.The combination of clustering and NN seems to be an effective solution because the other works applied it, such as (Chen et al. 2019) where firstly it executes an OPTICS clustering to extract trajectory, and then applies convolutional NN in order to classify the trajectory.(Nguyen et al. 2021) uses a probabilistic RNN-based representation of AIS tracks, and then a grid-based threshold to assess the anomaly of the vessel.The grid threshold allows the algorithms to adapt the global classified behavior analysis to the local route trend.
Also, the other approaches used within maritime surveillance to detect the vessel anomaly are: Fuzzy ARTMAP NN, Gaussian Mixture Models (GMM), Bayesian networks for false ship type, etc (Svenmarck et al. 2018).A very interesting approach to identify the vessel are the Dynamic Bayesian Networks (DBN) that analyze the traffic situations at sea and assess kind of relationship between them.Specifically, in (Anneken et al., 2019) this algorithm for identification of anomaly behaviors of vessels and reduction of unnecessary amount of data is elaborated according to the corresponding probabilistic model with graphical representation of Bayesian reasoning.In this approach, conditional probability is used with the time slices for random variables, that over time can obtain new attributes by passing from "parentinitial" to "child" situation.These changes can be abstracted as events with certain dependency rate, and if one event is realized, the others related will also happen in particular time interval.Applying this DBN to maritime environment and vessels as objects, the abstracted situations with random variables correspond to constituent events of vessel anomaly behavior as e.g.smuggling anomaly with particular attributes like position, vessel type, course, distance and approaching (Anneken et al., 2019) In this part, a soft information fusion and management toolbox and deployed in EFFECTOR project and then an Early Collision Notification System will be described.These AI features are used cinematic of vessels to take a decision.

A soft information fusion and management toolbox deployed in EFFECTOR project
Data and information fusion refer to a set of scientific methods and artificial intelligence algorithm to create or refine indicators by aggregating data from heterogeneous sources.More specifically in EFFECTOR project, the main function of fusion is enhancing situation awareness and reducing the number of information to be shared between different systems, increasing the global coherence of the information shared.Furthermore, as opposed to data, information embeds the context needed to be understood and interpreted.Within EFFECTOR, and for maritime safety in general, human operators are making decisions relying on the information they have access to.This is why we claim that the situation awareness of these operators should be improved thanks to semantic information, as it meaning is easily accessible to human operators.In this section, we describe the approach used in EFFECTOR for semantic information fusion.Specifically, in the project EFFECTOR is deployed a soft information fusion and management toolbox, InSyTo, providing core generic functions for high level information fusion (Laudy, 2010).It was used on several projects ranging from crisis management (Laudy et al., 2017) to investigation and oceanography, and we chose to use it in EFFECTOR for enhancing situational awareness and more particularly to detect meetings between several ships.The framework uses bipartite graphs and more specifically Basic Conceptual Graphs (Sowa, 1984;Chein and Mugnier, 2008) to represent information and knowledge.An ontology is used to adapt the toolbox for specific application domains.Basic conceptual graph are bipartite graphs containing concept and relation nodes.
The combination of core functions from InSyTo may provide advanced semantic information management functions.These core functions are depicted in Figure 2: Information Synthesis, Query and information fusion.The rectangle boxes represent concept node of conceptual graph and the circles represent relation of conceptual graph (between two concepts).The core functions of the toolbox are generic functions implemented over a generic maximal common subgraph (MCS) search algorithm.Depending on the way the MCS search algorithm is used, and on which parameters it is called, as illustrated in the Figure 2, several functions were developed such as information synthesis, information fusion, sub-graph fusion, information query, etc.To develop complex functionalities above InSyTo core function, one has to assemble them, and use them together with fusion strategies.
Fusion Strategies are domain and application specific rules used to provide the knowledge regarding compatibility of unit elements of the information graphs.Indeed, the fusion strategies are used to detect and fuse information items that are slightly different but describing the same situation.During an observation of an ongoing situation, these differences may appear from using different sources of information with potentially different level of precision or points of view.The main goal of sub-graph fusion is to detect and fuse compatible parts of two graphs.As opposed to information Synthesis (top of Fig. 2), however, the result of the subgraph fusion is only the common and fused part of the two graphs.One may see that, as the intersection of the two pieces of information.The information query function (middle of Fig. 2) can help to find all specified graph patterns within a Big Data graph.It is based on the search for injective homomorphism between the query graph and the data graph.The information fusion (bottom of Fig. 2) can help to find a specific situation model in observations.Within all the core functions of InSyTo, we added a traceability capacity (Laudy and Jacobé de Naurois, 2021).The aim is to keep records of all the fusion operations that were achieved on each unitary component of an information graph.The lineage graph records the initial source of each information item, as well as the succession of fusion operations together with the fusion strategies used.Adding this capability to the toolbox enabled us to improve the end user understanding and thus trust toward the overall system.For a specific use case, these different functions can be combined and specific strategy and similarity functions can be developed.
Also, trajectory fusion and abnormal vessel behavior is identified and managed in the EFFECTOR project using the InSyTo framework.More specifically, the suspicious vessel encounters are detected by a high-level fusion function,

Early Collision Notification System architecture and deployment
Collisions at sea pose a significant threat with potential serious consequences for human life, environment and economy and maritime safety in general.To avoid these effects in an effective manner and reduce the implications of an imminent collision, much research has been conducted to evaluate the collision risk (CR) of two approaching vessels.Based on the value of this index early notifications can be generated to help seafarers execute the International Regulations for Preventing Collisions at Sea (COLREGS) avoidance maneuvers in time.Researchers have proposed many CR evaluation methods including numerical [Liu and Liu, 2006] and fuzzy comprehensive models [Feng and Li, 2012;Xu et al., 2009], ship domain methods [Xu and Wang, 2014;Szlapczynski and Szlapczynska, 2017], fuzzy reasoning methods [Kao et al., 2007;Rizogiannis and Thomopoulos, 2019] and other.In the context of the EFFECTOR project, the Early Collision Notification service (ECNS) has been developed as part of the EFFECTOR Multi-level data fusion and analytics services for knowledge extraction and provision of enhanced situational awareness.ECNS aims at timely generation of notifications of imminent collisions between ships that could cause death at sea in the area of operation.In this way ECNS service contributes to an increased level of maritime safety by providing, at an early stage, alerts and the necessary reaction time to avoid vessels collision.The high-level architecture and the decision engine of the ECNS service are presented in Figures 3 and 4 respectively.Compared to existing research, the proposed service was built aiming to quickly discard pairs of ships that appear no collision risk and minimize the number of variables used as input to the fuzzy system in order to accelerate the decision process while at the same time achieve an efficient performance.
The input to the ECNS module is a rich set of data, containing kinematics information, (e.g.position, speed, course, turn rate, other) for the two most recently reported positions of both vessels as well as vessels' length and type.Using this input, many new parameters (e.Finally, in the decision system unit, a type-1 Fuzzy inference system (FIS) uses as input the set of variables (DCPA, TCPA, Relative distance) to evaluate the desired CR index where the membership functions (MFs) of both the input and the output variables are of the general form depicted in Figure 5.The paper discusses some of the most important recent AI capabilities based on Big Data sources, and applied in maritime safety and surveillance in order to enhance the overall cooperation and performance of inter/national agencies involved in the CISE network.We analyze the key features of AI approaches, that improve the maritime surveillance using AIS and other data, and that, according to the augmented data/information fusion processes and decision support tools, significantly contribute to the higher interoperability among maritime ICT systems and regional CISE cooperation of national agencies with purpose to enhance overall maritime safety.Specifically, the InSyto and ECNS tools deployed in EFFECTOR project concern the high level of development of AI-based fusion services for trajectory and movement tracking, necessary to detect vessel anomalous behaviour and assess the risk of possible vessel collision.Finally, we can conclude by saying that, maritime safety environment will achieve greater resilience and operational efficacy only by more intensive exploitation and combination of AI applications with advanced algorithms for vessel behaviour, risk events identification, assessment and control at sea and its timely, cost-effective exchange within CISE Network.

Figure 1 :
Figure 1: CISE Architecture and Information Flows supported with AI components Source: authors' adaptation Figure 2: InSyTo core Function Source: authors' adaptation

Figure 3 :Figure 4 :
Figure 3: The high-level architecture of the ECNS g. Distance to Closest Point of Approach (DCPA), Time of Closest Point of Approach (TCPA), Relative bearing, other), are calculated, as well as other useful intelligence (e.g.routes intersection point, determination of encounter type, ships approaching or surpassing).At the Rule based filtering unit, the speed, course, routes intersection point, and distance information are used to determine whether vessels are close to each other and approaching on a collision course.If both conditions are valid the processing flow moves to the decision system where CR is evaluated.Otherwise, the ECNS service checks the next pair of vessels.
. Case study: EU Project EFFECTOR A significant EU research and innovation project related to maritime surveillance strengthening is The End-to-end Interoperability Framework for Maritime Situational Awareness atStrategic and Tactical Operations (EFFECTOR).This project gathers national maritime safety and security institutions, vessel satellite surveillance and data exchange software integrators, RTOs and academia with the aim to foster collaboration among stakeholders, using a common Interoperability Framework for Maritime Surveillance and Border Security.Some of important methods and tools used to increase the situational awareness in maritime domain are the following: multi-layered data lake platforms, data fusion and analytics, knowledge extraction and semantics, collision notification, maritime ontologies and vessel surveillance AI modules operated through integrated C2 systems/platforms (SeaMIS, ENGAGE, MUSCA) and in full compliance with CISE and EUROSUR standards.These innovative technologies are deployed, tested and validated in three operational trials: France, Portugal and Greece[EFFECTOR Grant Agreement, 2020].Specifically, the end-user group, composed of governmental maritime safety and border authorities, provided relevant maritime data for Data Lakes collected from national LS for participation in the French, Greek and Portuguese Operational Scenarios and Trials, with final validation and evaluation of project technical solutions based on Key Performance Indicators.
Z.Paladin, N. Kapidani, Ž. Lukšić, A. Mihailović, P. Scrima, C. JacobÉ de Naurois, C. Laudy,  C. Rizogiannis, A. Astyakopoulos & A. Blum:Combined AI Capabilities for Enhancing Maritime Safety in a Common Information Sharing Environment 155 implemented to reason on vessel trajectory data generated by AIS systems.The InSyTo sub-graph fusion function is used to detect common sectors of different vessel trajectories.Moreover, application and domain specific similarity functions and fusion strategies are implemented to define what a vessel encounter is..A vessel encounter is considered suspicious if it lasts a minimum time duration and if the two vessels are at less at a defined geographical and temporal distance.A human operator further configures them to define the fusion conditions based on the specific application requirements.For the EFFECTOR case study, the InSyTo framework is connected to a Data Lake which contains vessels trajectories.After queries of vessel trajectories, the InSyTo framefork searches for encounter between vessels.Alerts in the CISE format are raised automatically to signal the beginning and the end of a suspicious vessel encounter, accompanied by the time and location data for each vessel pair involved.The two anomaly CISE types are "vessel approaching" and "vessel moving away".The InSyTo framework in EFFECTOR project is instantiated for automatically detecting risks and incidents and more specifically vessel encounter (or collisions).The goal is to enable better detection supports for operative agencies in maritime safety domain and efficient collaboration based on CISE network and architecture.