Vision-Based Maritime Flight: From Deep Perception to UAV Experiments

A UAV approaching a ship needs to know its position and orientation relative to the vessel. Waves, changing viewpoints, lighting, and limited access to reliable navigation signals make this a demanding perception and control problem.

FDCL’s maritime autonomy research connects deep visual perception, geometric estimation, and experimental validation, in collaboration with the U.S. Naval Academy. It builds from shipboard visual–inertial flight experiments to transformer-based perception and mixed-reality testing of a complete autonomous flight system.

The deep-perception work led by Maneesha (“Maneesh”) Wickramasuriya forms a central part of his doctoral dissertation, Deep Transformer Network for Autonomous UAV Launch and Recovery in Ocean Environments.

The foundation: autonomous flight from a moving ship

Earlier work by Kanishke Gamagedara, Taeyoung Lee, and Murray Snyder developed the onboard hardware, estimation, and control software for autonomous launch and landing from a U.S. Naval Academy research vessel in Chesapeake Bay. The experiments tested both RTK-GPS-based relative positioning and vision-based flight.

A delayed Kalman filter fuses measurements according to when they were acquired: a delayed observation corrects a past estimate, which is then propagated to the current time. This makes the estimation system account for sensing and processing delays while the aircraft and ship continue moving.

Learning to recognize the ship’s structure

The transformer approach learns recognizable ship structures from a single camera image. It predicts geometric keypoints on several parts of the vessel, recovers an individual pose from each part, and combines those estimates through Bayesian fusion. Using several structures reduces dependence on the visibility of any one ship component.

Collecting hundreds of thousands of labeled images at sea is impractical. Instead, the researchers reconstructed the vessel’s geometry and rendered approximately 435,000 synthetic training images, varying camera poses, textures, backgrounds, and lighting. The network learns ship structure across these variations rather than relying only on matching local image texture between successive frames.

From synthetic images to shipboard observations

The 2025 JGCD study evaluated the trained network on synthetic test images and real images collected during shipboard flight experiments. The real-image tests included normal, overexposed, and underexposed views.

For the three real-image datasets, position error was 0.66–0.97% of each dataset’s maximum range. These results demonstrate transfer from synthetic training to real maritime imagery under the tested conditions. They evaluate pose estimation from flight imagery; closed-loop flight using the transformer is addressed in the later study below.

Mixed reality: connecting perception to a flying UAV

The ICUAS 2025 study introduced a vision-in-the-loop environment using 3D Gaussian Splatting. Images captured around the research vessel are used to construct a photorealistic scene that can be rendered from new viewpoints.

The ICUAS 2026 study extends this environment to autonomous closed-loop flight with onboard perception and estimation:

1. A motion-capture system measures the physical UAV’s pose so that the renderer can generate the corresponding maritime camera view.
2. Those RGB images are streamed to the onboard computer, where the transformer estimates ship-relative position and orientation.
3. A delayed Kalman filter combines the visual estimates with high-rate inertial measurements, and a geometric controller uses the resulting current-time estimate to command flight.

The experiments demonstrated autonomous takeoff, trajectory tracking, and landing in the indoor maritime emulation shown in the opening video. The setup exposes perception latency, asynchronous updates, and onboard computing constraints while retaining physical UAV dynamics. It provides an intermediate validation stage before at-sea deployment of the integrated transformer-based system.

Complementary sensing and geometric estimation

The same emphasis on vessel geometry also supports LiDAR-based pose estimation. A point transformer learns 40 ship keypoints from sparse 3D scans, providing a complementary route to relative localization when image appearance is difficult to interpret.

Related work on invariant Kalman filtering for relative dynamics develops the geometric estimation theory for motion between two systems. This complements the demonstrated delayed-filter flight architecture and supports further research on ship-relative sensor fusion.

Related publications

• K. Gamagedara, T. Lee, and M. Snyder, Delayed Kalman Filter for Vision-Based Autonomous Flight in Ocean Environments, Control Engineering Practice, 143, 105791, 2024. Shipboard flight and delay-aware visual–inertial estimation.
• M. Wickramasuriya, T. Lee, and M. Snyder, Deep Monocular Relative 6D Pose Estimation for Ship-Based Autonomous UAV, AIAA SciTech Forum, paper AIAA 2024-2877, 2024. An earlier development of the deep monocular perception approach.
• M. Wickramasuriya, T. Lee, and M. Snyder, Deep Transformer Network for Monocular Pose Estimation of Shipborne Unmanned Aerial Vehicle, Journal of Guidance, Control, and Dynamics, 48(8), 1915–1930, 2025. Multi-part perception, Bayesian fusion, and synthetic-to-real evaluation.
• M. Wickramasuriya, B. Yu, T. Lee, and M. Snyder, Vision-in-the-Loop Simulation for Deep Monocular Pose Estimation of UAV in Ocean Environment, ICUAS 2025. Photorealistic maritime rendering and indoor testing.
• M. Wickramasuriya, B. Yu, J. Shin, M. Huslig, T. Lee, and M. Snyder, Hardware- and Vision-in-the-Loop Validation of Deep Monocular Pose Estimation for Autonomous Maritime UAV Flight, ICUAS 2026. Integrated onboard perception, estimation, and autonomous flight experiments.

Pose-estimation code · Doctoral defense announcement · Complementary LiDAR research