Overview
I am interning at Seasats, where I work on autonomous launch and recovery of unmanned aerial systems from unmanned surface vehicles.
The goal is to give Seasats’ autonomous surface vessels an eye in the sky. A drone rides on the boat, launches on its own, flies out to look around and identify objects in the water, and then lands itself back on the boat — with no human in the loop on either vehicle. The target grew over the summer: it started as a single drone launching from and landing on a Lightfish, and expanded into deploying a swarm from a QuickFish.
One of the problems was a launching and recovering on a moving platform. The deck the aircraft has to leave from and return to is a small boat that pitches, rolls, and heaves with the sea state, so launch and recovery have to be handled as a coordinated behavior between the two vehicles rather than as an aircraft-only maneuver.
Additionally, our drones do not have onboard compute. Putting the computer on the boat instead keeps each aircraft light, extends its range, and makes it cheap enough to lose. It also creates the two problems most of this page is about: there is video latency between the drone’s FPV camera and the ground station, and there is control latency, because every command reaches the flight controller through a USB radio link rather than a wire.
My Role
I worked on this as a software developer and owned the drone stack end to end: communication between Python and the aircraft, a Gazebo simulation environment to test and validate against, a custom launch sequence, motion planning, the computer vision — camera calibration, the landing marker, and tag detection — the precision landing algorithm all of that fed into, and the integration with the Seasats UI.
Talking to the Drones
PX4, the flight stack running on each aircraft, communicates over MAVLink. QGroundControl is the usual MAVLink client — a GUI for setting positions and parameters by hand — but the whole point here is that no human is doing that in flight. I used MAVSDK, which wraps MAVLink in asynchronous Python, so the boat-side computer can talk to the drones programmatically.
Almost everything custom happens in offboard mode, where PX4 hands control of the setpoints to an external computer instead of flying its own mission. That is the seam the launch sequence, the waypoint navigation, and the landing controller all live in.
Simulation
Having a simulator is always worth it before running anything on a real aircraft — every hour spent in sim raises the odds that the first real flight is uneventful. I used Gazebo, mainly because PX4 already integrates with it and runs the same flight code in simulation that it runs on the aircraft. A controller that behaves in sim is being judged by the same autopilot that will eventually fly it.
I built a boat platform with an AprilTag on its deck, driven by simulated wave motion, and flew a drone down onto it. That world is where the landing controller was developed, and where every change to it got tested before it went anywhere near hardware.
Launch
A drone stowed on a boat has to clear a hatch on the way out, and PX4’s normal takeoff command is not built for that. It climbs under closed-loop control at a rate the autopilot picks, which is smooth, well-behaved, and far too slow when what you want is to be out of the opening before the boat rolls.
So instead of sending a takeoff command, I put the drone into offboard attitude control and commanded thrust directly. That gives an open-loop punch straight up at whatever thrust I ask for, and the aircraft hands off to position control once it is clear of the boat.
Waypoint Navigation
Getting the drone somewhere is the simple part of the stack. It takes a latitude, longitude, and altitude, and flies a straight line to that point in the world as a stream of position setpoints. Everything harder — sweeping an area, staging over the boat before a descent — is built on top of this one primitive.
Precision Landing
This is where most of the summer went, and it took about a month to get right.
The target is an AprilTag landing board I designed for the deck. A single tag does not survive a whole descent: at altitude a large tag is the only thing detectable at all, and near touchdown that same tag overflows the frame and stops decoding. So the board carries one large tag alongside a set of smaller ones at known offsets, and the detector always reports the large tag’s center — directly when it can see it, and reconstructed from whichever small tags are still in frame when it cannot.
Before any of that means anything in metres, the camera has to be calibrated. I shot checkerboard images through the exact camera, resolution, and capture path used in flight, and solved for the intrinsics. The FPV lens is close to a 160° fisheye, so a pinhole model diverges badly and the fisheye model is not optional. A calibration taken through a different path than the one that flies is not a calibration.
Detection then produces a tag pose relative to the camera, and that is the input to the landing controller. The pipeline is a chain, and the shape of it is the part worth drawing:
Three findings did most of the work.
A downward camera beats a forward-slanted one. A tag directly beneath the aircraft is exactly where it needs to be readable at the end of a descent, and a slanted camera loses it precisely there.
Position setpoints beat velocity setpoints. Commanding velocity on top of PX4’s own position loop puts two controllers on the same axis, arguing. Handing PX4 a position and letting its loop do the work is what made landings repeatable, and it is why the diagram above hands off where it does.
Every camera frame has to be time-aligned to the flight controller’s state at the instant of capture, not the instant the detection finished. A tag pose is only true for the moment the shutter was open, and with video latency in the loop those are not the same moment. Fusing each detection against interpolated position and attitude at capture time — rather than against whatever the drone happened to be doing when the CPU got around to it — was worth more than making the detector faster. By mid-July the controller was landing 8 times out of 10, with the last half-metre the hardest part of the descent.
Swarm
Once one drone could launch, navigate, and land itself, we scaled to several. The idea that made it tractable was to stop thinking about the drones individually and treat the swarm as a single body: a commanded world coordinate moves the centroid of the formation, and each drone holds a fixed offset from that center.
Everything built for one aircraft then applies unchanged. A goto command sweeps the centroid toward the target and every drone follows its own offset across; the formation geometry becomes a purely geometric problem that never has to reach into the flight code. Separation between aircraft is a property of the offsets rather than something a controller has to negotiate in the air.
Future Work
The core building blocks all work — autonomous launch, live video, marker-based precision landing, and the command pipeline end to end. A four-drone swarm flew launch and landing on land, a two-drone swarm launched off a QuickFish, and we recovered a drone onto the QuickFish by hand. What is left is the list I would work down next.
- YOLO for object detection in the water. Giving the search leg something to actually find rather than just a pattern to fly. The catch is that a stock detector is a viewpoint classifier as much as an object classifier — a hull seen straight down is out of distribution however many pixels it covers — so this is a fine-tune on maritime imagery, not a threshold change.
- Test autonomous landing on the boat. The one thing we did not get to, purely on time. It is also the highest-risk unproven piece and the one the whole mission ends with, so it is the first thing I would fly.
- Build and test a larger airframe that carries its own compute. Putting the computer back on the aircraft removes the video and control latency that most of this page is about, at the cost of the cheapness that made losing one acceptable.
- Fly larger fleets. The centroid model was built so that adding aircraft is a matter of adding offsets rather than of changing the flight code, and that is the part worth proving out at scale.
Reflections
The biggest thing I took from this summer is that none of it was going to be reasoned into working. The schedule was really set by how many flights we got, so we tested early and often, and I stopped worrying about crashing. Every crash during testing is one that doesn’t happen during the run that matters. The trick is keeping a few beater drones around that you can rebuild in an afternoon, because what a crash actually costs you is the next few days of not flying. I also learned to freeze the code before a deadline. The change that makes it slightly better is also the change nobody has flown. And I also learned that simple is better, in both hardware and software. Every time this project got into trouble, it was because something had more moving parts than the job really needed. This was a really fun internship. I don’t think a lot of interns get this much ownership over a project, or get to work on something this cool. We picked our own drone parts, built the drones ourselves, and wrote the whole stack from scratch, and it ended up working. Hard to ask for much more out of a summer.