Computational Skies

Drones, autonomy, and aerial robotic saturation

This last year, I participated in a competitive autonomous drone racing competition. Starting from pretty much zero experience, I put an insane amount of hours and tokens into exploring the practice and training drones to fly through a course of gates and obstacles. The competition is hosted by the defense company Anduril and the Drone Racing League, and with that comes the implications.

In programming there are few holy grails, one of them being SLAM (simultaneous localization and mapping) and autonomous navigation. I wanted to learn more about training specialized AI robotic reinforcement learning models and see if I could figure out the autonomy stack. The competition's ultimate goal is to move into sensor-denied navigation (navigation without GPS or other external positioning systems), where a drone relies primarily on its IMU (sensors that measure acceleration and rotation) and camera to navigate.

Without GPS, it is a whole different game. Small IMU errors accumulate, and suddenly figuring out where the drone actually is becomes a problem of integration (calculating position from motion over time), filtering noise from sensor data, perception, and estimation. A camera is easy to imagine as a pair of eyes, but without a brain, it can't do much. Computer vision models (software that interprets images) like YOLO or MobileNet can be tuned to turn images into something useful, by interpreting landmarks as relative spatial coordinates. Then another system, intelligent or procedural has to decide what the craft should actually do with that information.

Using tools like Gymnasium (a reinforcement-learning simulation framework) and PPO (a reinforcement-learning algorithm) you can simulate millions of flights, but the gap between simulations and reality is brutal. Each improvement from hovering, navigating through one gate, and changing a sequence of gates are less incremental and requires conceptual overhauls like going from crawling to running a marathon. I made it through the first round of the competition by completing a "simple" course, but in the second I couldn't cross that sensor-denied gap. I am continuing the work now with my own research drone.

I saw this competition as an opportunity to deep dive into the drone and autonomy space. Throughout the project, I began thinking about the imminent point of robotic saturation and when machines begin to move themselves through the world without us.

Image from AI Grand Prix website

One of the safer ways to get into drones is in the digital space, using training simulators. The one the cool kids use is Liftoff. It gives you a feel for hovering, turning, and flying. It welcomes you with a nice Ukrainian flag and almost reminds me of the old Ace Combat games sponsored by the Japanese Air Self-Defense Force as a badge of legitimacy and who knows what else.

The dream of flight has always been something so enamoring yet so inhuman. It requires a totally different understanding of our bodies' spatial orientation and comes with its own chic, pseudo-sailing, jet-setting language. Flight sits in a funny space: unattainable and dangerous, even as we soar higher and faster every century. We are now crossing another event horizon: the miniaturization of aircraft and the introduction of hyper-controlled aerial systems.

Though colloquially known as drones, almost like a biological classification system, we can think of UAVs (unmanned aerial vehicles) as the family, multicopters (aircraft with multiple lift rotors) as the genus, and quadcopters (multicopters with four rotors) as the species.

The thing I found unintuitive about multicopters is the tight coupling of sensors and motor control, flight is built around constant correction or error. Instead of relying mainly on aerodynamic surfaces like a plane, the aircraft changes its rotor speed thousands of times as it tries to maintain a desired state—while the form can be dramatically different. Constraints are more tied to center of balance, rigidity, vibration control and organization. Inside the flight controller is essentially a continuous question "Where am I now, where am I supposed to be, and what force do I need to apply to close the difference?"

An observation of feedback and correction wanders into more esoteric territory: cybernetics (the study of control and feedback systems), cognition, and eventually the philosophy of mind. The drone senses itself, compares what it perceives against some desired state, acts on the difference, and tries again.

Once perception, positioning, planning, and control connect, the drone stops being only a remote-controlled aircraft and becomes something closer to an embodied agent (a system that can perceive and act in the physical world). This fuzzy edge of autonomy may in fact be the real inflection point.

Historically, flight has been a scarce commodity. Aircraft are expensive, conspicuous, centralized, and difficult to operate. A helicopter concentrates avionic capabilities: pilot, camera, communications equipment, lift, mobility, observation, sometimes cargo. Multicopters democratize and decompose that machine and add numbers.

We can begin thinking about these as distributed networks with different parts being coordinated drones, cameras, spotlights, relays. This effectively becomes a spatial mapping with cargo capabilities. The major step being human payloads which may remain exclusive to helicopters and airplanes until these machines are able to do it safely and economically.

Autonomy pushes flight further because it breaks a relationship that has limited aviation for more than a century: roughly one aircraft requires one person and often a team's attention.

With scale in technology and systems, we often need to shift from piloting to operating. If I want one remotely controlled drone in the sky, I need someone flying it. If I want one hundred, a hundred operators. But if the aircraft can navigate, avoid obstacles, coordinate, return home, and execute tasks themselves, a person can move toward supervising a fleet rather than manually flying each machine, more similar to a traffic controller than a pilot. At a certain point, the meaningful unit may be aircraft per person. This is a point when saturation becomes omnipresent.

The most important property of a drone may be presence over transportation.

Historically, aircraft exist primarily to move something through the air. A passenger plane moves people. A cargo aircraft moves goods. A missile moves a weapon. But a drone has value simply by occupying airspace. Its usefulness can come from remaining present above a place, observing it, communicating through it, or interacting with it over time. The drone becomes less like a vehicle and more like a mobile sensor and actuator occupying three-dimensional space. This is where the consequences become dystopian.

A fixed surveillance camera is generally immobile and sees what it is bolted in front of. A traditional surveillance network waits for a person to walk past a camera. An autonomous aerial surveillance network can send the camera to the person. It seems like only a matter of time before something like a Flock camera becomes true to its name and becomes a flying flock.

Police chases have been a staple of Los Angeles news coverage for decades, almost part of the city's environment. Recently I saw overhead footage of a pursuit through rural Oklahoma and the contrast was uncanny. It looked more like a scene from Starsky and Hutch. There was something cowboy about it.

What struck me wasn't the chase, but that this familiar omniscient aerial perspective had escaped Los Angeles. The camera in the sky was no longer dependent on a helicopter circling a major city. These events happen anywhere.

War has shown the other side of this transition in its most brutal form and is an inescapable reality in space; tragically, you cannot be a hobbyist without being subject to horrific imagery and are faced with the inherent moral dilemma of this technology's progress. The same stack that makes civilian autonomy useful—computer vision, cheap cameras, navigation, networking, object tracking, autonomous landing, resistance to electronic warfare—also creates extraordinary asymmetric force.

A relatively inexpensive aircraft can destroy a machine worth orders of magnitude more, or simply force an opponent to spend enormous resources trying to detect and stop it. There is a strange asymmetry here: thousands of dollars in drones can demand millions in counter-drone infrastructure. The 2026 World Cup offered a civilian version of this dynamic, with enormous resources devoted to detecting, tracking, and intercepting small aircraft that themselves might cost only a few thousand dollars. The United States allocated $250 million toward expanding these counter-UAS capabilities ahead of the tournament, while unauthorized drone activity could carry fines of up to roughly $100,000.

Yet the basic stack can be almost identical in completely different circumstances: perception, localization, planning, communication, control, and a payload. The aircraft itself is almost morally empty. As with most things the consequential question is who gives it a goal.

Agriculture is becoming one of the less dramatic but perhaps more transformative versions. Drones can inspect crops plant by plant, spray targeted areas, count livestock, locate irrigation problems, identify disease patterns, and monitor enormous areas with very little human presence. The result may not look like a spectacular flying robot. It may look like an invisible layer of automation continuously moving through space and collecting information. That is probably closer to what an industrialized aerial drone future actually looks like.

The individual use cases are interesting, but the larger question is what happens when there are millions of these machines. What happens when flight becomes cheap and autonomous enough that the sky begins behaving more like network infrastructure (a shared system continuously providing information and services)?

Who has the right to occupy the air twenty feet above a street? Who gets to watch from there? How much noise is acceptable? Who controls landing locations? How do autonomous aircraft negotiate right of way? What happens when somebody hacks one? Who is liable when one makes the wrong decision? What happens when every police department, utility company, logistics provider, construction site, farm, security company, and private citizen wants access to the same three-dimensional space? The thing is, we may normalize all of this very quickly.

A modern airport would be almost incomprehensible to someone living two hundred years ago. Thousands of people inside enormous metal aircraft descending from the sky, guided by invisible radio instructions and precisely coordinated schedules. Drones inherit the language and add a robotic uncanniness to it.

For most of human history, we reserved something capable of hovering above you, observing you, descending from the sky, delivering a message, protecting you, pursuing you, or appearing in groups as angelic or supernatural. The watcher, messenger, guardian, eye in the sky.

Drones can be playful and horrifying, mundane and futuristic, lifesaving and invasive. They combine the sexiness of aviation with the uncanniness of robotics, the unease of surveillance and the logic of automation.

The quadcopter itself may only be a temporary embodiment. The UAV stack is more pure to the concept. Future autonomous aircraft may have wings, rotors, ducted fans, organic forms, lighter-than-air bodies, or shapes we don't currently associate with aircraft at all.

As a substrate this technological transformation is more important than the form factor. The sky is becoming computational space, realizing where it is and where it is going.

Rob Renn (future Rob)