In order to be able to realize fully or partially autonomously operating watercraft, the reliable, detailed and long-range recording of the current scene as well as the robust state estimation and classification of objects are of considerable importance. Radar sensors, which are almost exclusively used in the maritime context, do not provide the required quality and detail. Instead, optical sensors combined with LIDAR (see figure above) are ideal for creating a suitable image of the complex dynamic 3D scene and then making it available to tracking and classification algorithms. Within the project, this problem will be addressed by an interdisciplinary team consisting of staff from the Institute of Optical Systems (IOS) and the Institute of System Dynamics (ISD) in the following ways:

• Based on a multi-sensor system (lidar coupled with multiple cameras), the environment is captured at any point in time as a set of unregistered 3D point clouds and color information. This data will serve as input to a deep-learning architecture to be developed as part of the project. In contrast to the convolutional neural networks used today, these neural networks must be able to cope with the measurement points typical of 3D point clouds without fixed neighborhood relationships.
• With the help of these networks, objects within the point clouds of the respective sensor can be detected and classified in real time and sensor-specific depth maps can be generated. At this point, the resulting object lists and depth maps are neither fused between the individual sensors, nor across the individual time steps. This fusion does not occur until the next step. Because of this very late fusion, the multi-sensor system can be built in a modular fashion, which at the same time allows for a very robust fusion based on the individual sensor data already heavily pre-processed by the network.
• Using methods for tracking multiple extended objects, the object lists and depth maps of the individual sensor modalities are fused so that the detected objects can be classified, and their motion and orientation changes can be tracked in a uniform reference system over time. Due to the large number of different objects and motion patterns, multi-model approaches are required here. Alternatively, it will be investigated to what extent parts of this task can already be solved within the neural network architecture.

The sensor system for autonomous environment sensing on inland waters developed within the proposed project will be installed on the already existing autonomous watercraft and the water robot of the ISD. Two exemplary applications for collision avoidance and automated docking are envisaged. By means of a combination of optical, systems engineering and machine learning methods, the optical environment detection is to be performed on the basis of multiple non-registered sensors. The close proximity of the project participants and associated partners Catamaran-Reederei Bodensee, ZF Friedrichshafen AG, Konsortium Autonomes Fahren and Robert Bosch GmbH to Lake Constance offers optimal conditions for a successful implementation of the project.

Autonomous docking attempts

After fundamental reconstruction of the research boat Solgenia of the Faculty of Electrical Engineering and Information Technology, she could be launched again in summer 2020. The conversion was carried out with a view to future viability for research in the fields of autonomous systems and alternative propulsion systems, largely under the direction of the ISD-Konstanz. Already in December 2020, the first fully autonomous system maneuvers could be successfully tested (see videos). The videos show docking tests in moderate swell in the upper row, and in strong wind in the lower column. Algorithms developed at the ISD Konstanz were used here.