Mooring a boat is a complex operation even for experienced navigators. Poor visibility from the helmsman’s position, blind spots, the difficulty of seeing obstacles and perceiving their distance from the boat, make the maneuver difficult, long and risky.
The solution that EVS developed uses a set of cameras mounted on the mast to observe the boat surrounding area. The system processes video streams from cameras by merging and geometrically aligning the views into a single view from the top (“bird-eye view”) that allows the operator to see 360° all around the boat. In addition, the system is able to perceive the presence of obstacles emerging from the water surface around the boat and to indicate, in overlay, their position to the operator. This display mode provides the operator with all the information necessary to safely carry out the maneuver, similarly to how we are used when parking our cars equipped with a parking assistance system.
From the end user point of view, this system increases the maneuver safety, especially for less experienced operators, or those who cannot count on the help of other crew members.
For boat builders, instead, this system may represent a chance to enlarge the user base for same vessels, including potential customers that love sailing, but have limited experience and therefore the need for assistance in delicate maneuvers.
ZF Padova Srl, leader in the production of marine propulsion systems. It supplies systems and components for all types of ships, motorboats, high speed boats, work boats and commercial ships, to boat builders, shipyards to engine builders all over the world. It is a company belonging to the ZF Group.
The developed docking assistance system combines two functions in one:
- the generation of a 360° orthographic view of the boat by merging images from multiple cameras (“surround view”)
- the detection of obstacles around the boat indicating their presence graphically on a virtual bumper (“obstacle detection”)
The project required several computer vision skills such as the 3D reconstruction of the environment starting from a moving multi-camera system, the optical and perspective image correction, the “stitching” of the images, the brightness and color correction, the calibration of the parameters of the entire system.
One of the most challenging aspects is due to the presence of water whose appearance depends on many factors such as light, wind, environment, seabed and clearness. The water ripples, froths, gets dirty, reflects, and shines through. This sometimes makes difficult to distinguish water from obstacles. Thus, advanced machine learning techniques have been adopted to “teach” a neural network to make this distinction and to do it well even when the domain changes.
We selected the most suitable cameras and studied their optimal positioning by working in simulation and 3D modeling environments.
We chose a state-of-the-art computing platform, that was the most suitable in terms of performance, reliability, consumption, size and costs. It is a hardware module based on multicore architecture with DL acceleration and necessary multi-channel video camera support. After an initial development and validation of the algorithms in a PC environment, all the software was ported to the selected embedded platform (“embedding”), optimizing it until achieving the required performance in terms of throughput, latency, and accuracy.
The system was then tested on a real sailboat giving promising results both in terms of performance and usability in assisting the docking maneuver.
The project was carried out in close collaboration and with the support of the ZF Marine team.