VERIFICATION OF THE IMAGE PROCESSING SYSTEM IN REAL CONDITIONS

 AVAL – Autonomous Vessel with an Air Look, is a research project that aims to develop autonomous navigation of ships. The system uses three independent sources of information i.e. radar, AIS – Automatic Identification System and cameras, which can be located on a drone or ship’s superstructure. The article presents the results of testing of an image processing system in real conditions on m/f Wolin.


INTRODUCTION
AVAL technology consists of three components: 1. Maritime Autonomous Surface Ship (MASS) technology. The heart of MASS is an anti-collision component and a system of communication with Unmanned Aerial Vehicle (UAV).
2. UAV technology supports the marine navigation system in a collision situation. UAV is equipped with a camera and sensors to register and transfer video and hydro-meteorological data.
3. Image Processing System (IPS). A key element of IPS is an algorithm for the detection and recognition of dangerous objects at sea (such as leisure craft, icebergs, whales, etc.) in images recorded by a UAV camera.
Attempts to build autonomous ships have been undertaken in many countries and for various reasons. The most important ones are to enhance safety and efficiency of sea shipping. These include the shortage of suitably qualified navigators. The level of advancement of the work vary, but in all cases the studies are limited to internal waters. The scope of navigation limited to internal waters is crucial here, due to the complete lack of provisions enabling the operation of an autonomous ship in international waters. In practice, possibilities of conducting any experimental works related to autonomous vessels in waters accessible to everyone are limited. Navigation of autonomous vessels in open waters will most likely require the creation of a system for monitoring the movement of such ships from a landbased centre. Besides, training will be needed to prepare new specialists to supervise the navigation of such ships.
The present shortfall of seamen is notable. Estimated at 16,500 (2.1%), it is expected to rise by 2025 to 147,500 skilled workers [1]. This is the social part of the challenge.
Research on technologies for autonomous vessels is conducted in a number of centres around the world. Technological solutions should fulfil requirements concerning safety, security, reliability and efficiency of shipping. The challenges are summarized in the article "Why we will never see fully autonomous commercial ships" [2]. The main points are cited below: 1. "One existing obstacle for automating larger vessels is battery technology." 2. "Even more challenging obstacles to the success of autonomous vessels will be the expense and complexity of designing such systems." 3. "Autonomous vessels would save money by not having a crew, but ship-ping companies will in many cases be simply replacing merchant mariners with other workers, most likely more expensive technical workers, who will work in offices on land or will be on call to assist autonomous ships across the oceans..." 4. "The weather, wind, waves, fog, obstructions, marine mammals, saltwater, birds and other ships that vessels encounter mean an autonomous ship will require incredibly complex technology able to withstand the chaos of the ocean environment and enable a ship to respond remotely to any incident or emergency." 5. "The most serious concern regarding autonomous vessels is the one that will very likely keep them from ever being employed: the risk of exploitation by adversaries, hackers, terrorists, criminals, and other malign actors..." The concept of autonomous shipping is becoming more and more popular nowadays. The number of projects commenced , which started last years, shows also, that this strong trend will increase. Although all projects mentioned below are focused on autonomous ship, each of them has its' specific. Project MUNIN [3] was one of the first concepts, while projects Milli-Ampere, Ballstad and ASTAT [4][5][6] focusing on short-sea shipping, i.e. ferry shipping in particular. Hull to Hull [7] is dedicated to close proximity navigation, while Autoship [8] focuses on the target detection. Yara Birkeland [9] and AAWA [10] are dedicated to de-signing and building full-size ships. AVAL [11] is one of the first, who uses drone technology for the detection of targets at sea.
The main goal of our research carried out in real conditions was to identify potential gaps in the algorithms developed for the purpose of IPS and tested in laboratory conditions.

I. IMAGE PROCESSING SYSTEM
An average of 1,390 containers has been lost at sea each year, according to a special survey of the world's ocean carriers conducted by the World Shipping Council [12]. There have been many collisions between yachts or small ships [13] with floating containers. There were 292 reported collisions with whales over the past 27 years. A lot of whalevessel collisions are also reported to the B.C. Marine Mammal Response Network [14] hotline and investigated by the Fisheries and Oceans Canada every year. However, the involvement of larger vessels is likely underreported, as they are less likely to detect a strike. The Whale Detection Initiative will help protect whales in Canada's waters as part of the Oceans Protection Plan. This initiative will provide $9.1 million over 5 years to develop and test technologies to detect whales in near-real-time on both the west and east coasts. The early detection will help prevent vessel collisions with whales and protect these marine mammals. One database of ship collisions with icebergs [15] de-scribes 560 incidents and collisions between ships and icebergs or floating ice. Only in 2017, the US Coast Guard [16] counted 4,291 accidents that involved 658 deaths, 2,629 injuries and approximately $46 million dollars of damage to proper-ty as a result of recreational boating accidents. 65% of these accidents were caused by a collision with commercial vessels or recreational boats. Despite advanced navigation equipment such as GPS, echo sounder, radar, AIS, electronic charts, no device available today can detect such potentially dangerous objects, particularly in real-time. Due to the inability to mark such objects, technologies based on maritime video processing are one of the most promising solutions to solve the challenge. The image processing system (IPS) prototype is an ICT system consisting of a specially designed device and software. We have called the product version of the system IPS (Object Recognition and Classification Awareness). The device, whose main part is a camera, captures an image of the area around the ship, which is then processed by a set of algorithms for detection, classification and geolocation of objects dangerous for ships. IPS "sees" a lot more objects. IPS detects, classifies and geolocates in real-time such objects as whales, containers that have been lost by ships, drifting icebergs and plastic recreational craft without an AIS. IPS can be applied in today's ships and future vessels with semiautonomous and autonomous systems. The detection of objects invisible for standard navigation devices is a key challenge of autonomous maritime navigation systems because such function can replace visual observation conducted by the navigator.
None of the existing products can automatically classify and locate objects hazardous to ships and yachts at sea. There is no known marine vision system that processes the image in terms of detection, classification and geolocation of objects that can be integrated with a marine navigation system, which automatically generates manoeuvres and/or anti-collision trajectories. Currently, several R&D projects with goals similar to that of IPS have been initiated, but the results of these projects have not been used in a commercial product yet. There are no direct competitors as none of the existing products has a full set of IPS features. The current solutions may ensure detection and classification of objects but in different classes, without geolocation of objects and data fusion with other navigation data.
IPS monitors an area around the vessel (up to 10 Nm) using a special device with cameras, which records an image and runs an AI-based algorithm. It detects, classifies and geolocates potentially dangerous objects in real-time (very important). There are many recognition and classification systems on the market for any kinds of objects such as human faces or cars. The image registered in the marine environment generates specific and serious challenges. These challenges can be grouped in four blocks: a) horizon detection when the camera is installed on a mobile platform, b) registration -the situation where different frames in a scene corresponding to the same physical scene with matching coordinates, c) water background subtraction which changes continuously both in spatial and temporal dimensions due to waves, wakes, foams, and speckle in water which are inferred as foreground by a typical background detection method, d) foreground object detection -since the general dynamic background subtraction and fore-ground tracking problems do not require the detection of static objects, no integrated approaches exist that can simultaneously detect the stationary and mobile foreground objects. This is an open challenge for the maritime scenario. Current research on object detection in images may be applied for detecting objects in individual images, thus catering for both static and mobile objects. However, the complex marine environment with the potential of occlusion, orientation, scale, and variety of objects make it computationally challenging.
Therefore, unique solutions that address these challenges are needed and so are algorithms with better adaptability to the various conditions encountered in the maritime scenario. Thus, the field is rich with possibilities of innovation in maritime video processing technology.
A significant challenge of the IPS system is the hardware layer for visual data acquisition. Specialized optoelectronic apparatus for observing the surroundings of the ship must be properly oriented and stabilized. The compensation of the ship's pitch and roll must be done in real-time and the angular values deviating from the level known. The narrow viewing angle, if necessary, must be supported by the physical rotation of the observation head in a determined manner. The position of the observation head resulting from relative movement must be provided to geolocation algorithms to correctly calculate the position of the detected obstacle relative to the ship.
IPS provides critical information for anti-collision manoeuvres during the entire voyage of the ship or yacht: 1. Detection of static and dynamic objects within 10 Nm from a ship or yacht (presented range for vision sensor mounted 24 m above sea level): The system continuously monitors the surroundings of a vessel and at the same time detects, classifies and locates visible objects. This improves the navigator's ability to spot dangerous objects around the ship. Thanks to the warning, the time to make a decision about the anti-collision manoeuvre is extended.
2. Classification of detected objects in the following categories: ship, yacht, iceberg, whale, container, within 5* Nm: 1) to determine the risk of collision with a specific object depending on its category and location; 2) to verify if the navigator can see all objects through binoculars.
3. Detection, geolocation and classification of the objects within 2 Nm (presented range for vision sensor mounted 24 m above sea level): to include accurate data about the category and geographical coordinates of the object in the anti-collision algorithm, which automatically generates an anti-collision manoeuvre or anti-collision trajectory. 4. Self-tuning of the image recognition algorithm: selftuning improves the efficiency of detection, geolocation and classification of the objects. Self-tuning is applicable to repetitive routes -such as those of ferries.
5. Data fusion with AIS, ARPA (radar) and digital charts: to integrate IPS with maritime navigation systems.
Because of that, it will be possible to opt-out of navigator's visual observation, the key function of autonomous ships in the future.

II. RESEARCH
Currently, the image processing system IPS is not integrated into the navigation system. The next goal of our research is the integration of IPS with the real operating marine navigation system. NAVDEC is the system chosen for first integration attempts (Fig. 1. For this purpose, the fusion data algorithm (navigation data with image processing resultant data) will be designed, implemented and tested in real conditions. On the following diagram the proposed diagram of integration IPS into navigational system is presented (Fig.2).

Fig. 2. Diagram of the IPS components with the schema of an integration with maritime navigation systems
Tests were carried out on m/f Wolin in August 2018 and February 2019. During the AVAL project, two main types of IPS devices were considered and tested: installed on custom designed drones hovering over the ship and installed on the top navigation bridge (mounted for example 24 m above sea level -observation deck on ferry Wolin -see Fig.3.). Finally, we have decided to develop the IPS technology as a stationary solution (without drones) due to many legal and technical barriers resulting from the use of drones for continuous video observation at sea.
During the tests, the camera was mounted on the bearing deck, on the port side of the ship. Materials collected from the camera were used to develop a method of determining the position relative to the ship visible in the camera image objects. The following table shows the main features of the IPS technology and navigational challenges that are solved by image processing algorithms collected by the vision device.  Fig.3.) **The ability to detect and classify an object, depends on its size and distance from the ship or yacht. The value of the parameter is calculated under the assumption that the measured dimension of the object is parallel to the camera's projection plane.
The projection of the real 3D points into the image is obtained by a perspective transformation.
The position of the camera in its coordinate system is shown by the rotation matrix R and the translation vector t. Matrix rotation can be obtained by the product of rotation in individual camera axes (Fig.4):

Fig. 4. Rotation matrices [17]
Object identification analysis was carried out by comparing the results of the algorithm to a certain one knowledge base defined by a human expert. This method is described in the literature as "ground truthing" and can be described as follows: 1. First, prepare the Ground Truth by writing down information about each frame in the tested sequences. This information includes the number of objects in a given frame of the movie and rectangles indicating the location of these objects 2. Generate a result file for the ART program that lists the objects detected by the SPO algorithm in test sequences (objects also have the areas they are located in) 3. Load the GT file and the resulting ART file and then: 1. For each GT object, look for the object that intersects it (in fact, the specifying rectangle the area in which it is located) in the output of the SOP algorithm 2. If the common area of the rectangles is greater than 50% of the area of GT, we qualify the object as detected (if there are more hits, only the biggest hits anyway). In some cases we lower the criterion to 40 and 30% due to the fact that the detection was still correct (in total less than 10 frames).
3. No hit goes to the exit as 0% of the common area 4. Calculate statistics based on the number of hits versus the number of all objects in GT (hits + no) The results of the analysis of the individual sequences are shown in the table below.

ORCA functi ons
Problem solved Detecti on of static and dynamic obj ects, 20 m l ong ** and placed withi n 10 * nauti cal mil es (Nm) from a ship or yacht.
The system conti nuousl y moni tors the surroundings of a vessel and si mul taneousl y detects, cl assi fi es, and l ocates vi sibl e objects. This i mproves the navi gator's abi li ty to identi fy hazardous obj ects around the ship and make decisions regarding appropriate anti-col li sion manoeuvre options.
Classifi cation ** of detected obj ects wi thi n 5 Nm into the foll owi ng categories: shi p, yacht, i ceberg, whale, contai ner.
Determine the ri sk of coll isi on wi th a speci fi c object based on its category and locati on. Veri fi cation of the navi gator's abi li ty to see and i denti fy all objects through binoculars.
Geol ocation of the detected objects withi n 2 Nm * .
Provisi on of accurate data about the category and geographical coordi nates of the object to the anti-coll isi on al gori thm, whi ch automatical ly generates an anti-col li sion manoeuvre or anti-col li sion traj ectory.
Sel f-tuning of the image recogniti on al gori thm Self-tuni ng improves the efficiency of detection, geolocation, and classificati on of the objects. Sel f-tuni ng is appli cable for repeti tive routes -for example i n the case of ferri es.
Data fusi on wi th AIS, ARPA (radar) and digital maps Enhancement of mariti me navi gati on systems with ORCA' s real-ti me data can reduce the need for the navi gator's eye observati on -the key function of autonomous shi ps i n the future.

III. CONCLUSIONS
As can be seen in the results of the analysis, the total number of identified objects in the test sequences exceeds 85%.
In addition, the quality of object recognition was also analyzed on the basis of determining the recognition confidence provided by the algorithm during the analysis of test sequences. A detection was counted as the positive one only when the predicted area of the intersection was greater than 50% of the area of the un-ion of the ground true box and predicted box (IoU -intersection over union). Since in many cases the same object was properly detected in one frame and not detected in the next one, a simple heuristic assuming that if IoU is near 100% in one frame there is a high probability that the object will still be present in the next one at the same place would significantly improve the results.