The drone industry is emerging in every sector of the economy. Drones are being used to help agriculture, videography, journalism, building inspections, and search-and-rescue missions. But it isn’t just this new piece of technology and hardware that is bringing new opportunities to these industries—it’s software.
“The drones are not interesting by themselves; what is interesting about drones is what they can do: the data. What they can do will be defined by software,” said Chris Anderson, CEO of 3D Robotics (3DR), a drone manufacturer and technology provider.
The drone industry comes with huge opportunities for businesses and drone operators, but developers have the responsibility to turn the technology into something that will be useful and financially beneficial to businesses while also ensuring safety. “As technology matures, drones will increasingly be defined as ‘laptops and GPUs that can fly’ instead of ‘aerial platforms that can take pictures,’ ” said John Laxson, software engineering manager for Kespry, a provider of automated drone systems.
(Related: Why drones are such a big deal for developers)
Developers will not only be tasked to develop software capable of collecting, tracking, analyzing, managing and making sense of data, but also making sure these unmanned vehicles can securely operate in the sky.
“The future of this industry will be driven by software, and the opportunities are abundant. We are just getting started,” said Chad Sweet, director of engineering at Qualcomm.
Those opportunities range from building agricultural solutions that can help farmers measure their crops and monitor vegetation, creating simple-to-use interfaces so operators can focus on the task at hand, to developing a system that can help rescue efforts in emergency situations, and even help firefighters control wildfires.
“From a software developer’s perspective, this is a golden age of development on these platforms,” said Anderson. “We put the software in a box, and that box is called a drone, but all the value lies in the software.”
Software is the brain of the drone, and without it the technology wouldn’t be able to fly, added Carley Hopkins, product manager for Aeryon Labs, a drone developer and manufacturer. “As [drones] continue to evolve, engineers are being pushed to bring new technology such as obstacle avoidance and computer vision to smaller, lower-cost platforms,” she said.
Developing for drones
We are just starting to scratch the surface of drones, and with the government beginning to embrace the industry, a lot more use cases are opening up, according to Kespry’s Laxson.
“There’s everything from the autopilot actually managing the rotors and keeping the drone in the air, planning the path the drone will take, collecting useful data from onboard sensors, to getting it all back to the cloud platform, making gigabytes and terabytes of images useful and feasible to collect and analyze in the user’s browser,” he said.
Developing software and applications for drones is not radically different from developing software for mobile devices, according to Daniel Bowen, an engineer at Silent Falcon, a provider of drones and drone solutions. “As far as doing the UI for controllers themselves, that’s not really a whole lot different,” he said. “In fact, some of the UIs run on mobile devices, so you can have a ground station where the ground station is basically your iPad.”
The most popular programming languages developers are using to create drone software are often C or C++ because of their flexibility and low-level control, according to Bowen.
“However, if you want to run an app on top of the flight code via APIs, Python is kind of encouraged,” said 3DR’s Anderson. “You could write in C or Java, but Python is the go-to language for higher-level functions.”
In addition to the already familiar mobile sensors such as accelerometers, GPS, magnetometers and cameras developers are already working with, there will be more opportunities to leverage sensors in drones. Some of those new sensors include barometers, Lidar, sonar, chronometers, infrared sensors, thermal cameras, spectrometers, and gyroscopes.
“What we do with our eyes, drones and other mobile robots have to do with everything,” said Anderson. “Their eyes are not as good as ours and they are not built to analyze visual information like ours. They have to augment visual with other kinds of data, like [from] laser and radar.”
AI and machine learning
The software development industry has already been advancing computer vision and machine learning capabilities, and those advances and techniques also apply to drone development, according to Qualcomm’s Sweet. He explains that drones will need to leverage computer vision and machine learning in order to become a fully autonomous system.
Development gets tricky when you start getting into the flight code of the drone, according to Laxson. “At a low level, there’s a safety issue in drones that doesn’t exist in more types of software,” he said. “Crashing an embedded processor, or even a slight change of controller variables, could result in the drone falling out of the sky.”
The flight code designed to actually fly the vehicle is located on an embedded system, and it controls the motors and rotors. “Usually you have an embedded system that drives the plane, and the UI manages and oversees whatever is going inside the plane,” said Bowen.
Typically, developers won’t have to worry about the autopilot or flight code involved in the drone because the manufacturers creating the technology aim to take care of all the of hard components in order to make sure a developer isn’t going to crash it, according to Anderson.
“It is a little bit like if you write a bad app on your iPhone. The app might crash, but you are not going to kill the phone because Apple won’t let you,” said Anderson. “As manufacturers, we as an industry black box most of the complex parts of the drone.”
Aeryon’s Hopkins suggests developers decide what their strengths are, whether they are in embedded software, application software or Web development, then figure out how they will help them be a part of this new field.
“Like any new industry, there are lots of exciting new challenges. The UAS [drone] industry is particularly exciting because there are opportunities in a variety of software engineering roles,” she said. “There are applications that run on the aircraft itself to process all of the data and control the sensors, applications for the user to interact with the aircraft, and applications for pre- and post-flight activities such as stitching the images together to create a 2D map or 3D model.”
How the regulations affect developers
The future of the drone industry lies in the commercial sector, according to 3DR’s Anderson. The Federal Aviation Administration (FAA) is currently working on commercial drone regulations that will help safely implement drones into the national airspace. Currently, drone operators who are granted Section 333 exemptions and have a UAV operator’s certificate are able to fly drones less than 55 pounds up to 500 feet, within their visual line of sight at up to 100 mph away from people, cities, airports and other restricted airspace.
The FAA is also looking at creating regulations that would allow micro drones to operate over people, and is conducting studies into beyond visual line of sight operations. This legal landscape directly impacts developers, according to Qualcomm’s Sweet. “Regulation constraint demands result in less motivation to be creating these types of applications,” he said.
But developers can play an important role in advancing and helping the regulations. Eventually companies hope to fly over people and cities, and to deliver goods via drones. For this to happen the government needs to ensure these drones won’t harm anyone not directly involved in the their missions. To do this, the drones need to be able to sense and avoid objects.
“The FAA is very concerned about hurting its very reliable airspace, and rightfully so,” said Sweet. “There is a lot of opportunity for software to help drive the future, certainly in aspects of safety, security and intelligence.”
Ensuring drones are able to sense and avoid objects is not only a hardware manufacturer’s problem, it is also a task for software developers, according to 3DR’s Anderson. “The sensor has to be integrated into the drone some way, and that is typically through the manufacturers, but then making sense of the data from the sensor is all about software,” he said. “The whole sort of computer vision, machine learning and sense and avoid problem is going to take the best software engineers of our generations decades to solve.”
In addition, the FAA is expected to release Part 107 commercial operations rules soon, which will no longer require operators obtain a manned aircraft pilot license. This puts the responsibility onto the developers to create simple and easy-to-use user interfaces that make flying drones as easy as possible.
Drones and the robotic operating system
When the Robot Operating System (ROS) project, a collection of software libraries and tools for developing robot applications, started about nine years ago, the commercial drone industry wasn’t even a thought.
“ROS 1.0 was not designed for drones, it was designed for indoor robots,” said 3DR’s Anderson.
Despite the lack of initial use cases designed for drones, the industry gravitated toward ROS for its work in robotics, its proven success with GPS autopilot, and its ability to test safely. “ROS has been demonstrated to be the premier software platform for developing robotics applications in a wide variety of domains, from indoor service robots to agricultural robots to industrial robots,” said Brian Gerkey, CEO of the Open Source Robotics Foundation (OSRF), a organization who manages ROS. “ROS similarly provides drone developers with huge benefits in terms of access to tools, algorithms, and a global ecosystem of scientists and engineers.”
With the drone industry taking more and more interest in ROS, and already offering support for ROS through built-in and optional add-on solutions, the OSRF has decided to make drone development a priority in its upcoming release of ROS 2.0. “We’re making sure in ROS 2 that we support this domain even more so because drones are one of the most important areas of robotics today,” he said.
According to Anderson, who is also the chairman of Dronecode, ROS is now a part of Dronecode, and with version 2.0 expected to be designed more for mobile devices like drones, developers can expect more high-level advances from the project. “ROS is very much part of the future, especially for the higher-level stuff such as machine learning, swarming, and computer vision,” he said.
According to Gerkey, in order to improve its suitability for drone applications, ROS 2.0 will include improved support for low-power embedded computers and microcontrollers, the ability to develop code with real-time execution requirements, and enhanced communication over low-quality or intermittent wireless links services.
“ROS 2 is the next big step for us, and we expect that it will give drone developers even better support for building their dream applications,” he said.
ROS 2.0 is still a work in progress with alpha releases being made available about every six weeks. “Each new alpha release has greater functionality, but it’s not yet at the point where someone would want to base a new project on ROS 2. We’re aiming for a beta release that would serve that purpose later this year,” he said.
Where to get started
As the drone community continues to grow, a variety of resources and tools are being made available to help developers start digging into drone development as well as to help new projects and solutions soar.
“One beautiful thing about the state of the industry is that it’s already possible to buy a drone or assemble one from parts and control it via a simple SDK right from your laptop. Where you’re curious or passionate, you don’t have to invest a ton to play around with the technology,” said Laxson.
According to drone experts, the top resources drone developers should look at are:
Dronecode: An open-source, collaborative project managed by the Linux Foundation. The project aims to provide developers with resources and tools to help them developer better, cheaper and more reliable drone software.
DIY Drones: A drone community that provides resources for building drones both from a hardware and software perspective.
DroneKit: An open-source drone app development platform from 3D Robotics. DroneKit provides developers with an SDK and Web API to help them get their apps off the ground.
DJI SDK: DJI provides SDKs to help developers create drone apps. The company features a mobile SDK, an onboard SDK, a guidance SDK, as well as SDKs for Android and iOS.
Parrot AR SDK: Parrot provides resources for developers to start learning the basics of drone development. It features touch and tilt flight controls, live video streaming, video recording, and photo shooting.
In the future, Laxson hopes the community will be able to develop a common framework designed to understand the drone’s environment so developers can focus on the data.
Coming into the drone industry
While there are already a number of companies and developers that are building their businesses and careers around drones, there are some that are looking for ways to incorporate drones into their already existing strategies. For example, Qualcomm has been known as a provider of processing power and technology for smartphones and mobile devices. As the drone industry began to emerge, the company decided to figure out more ways its technology could expand into the new market.
“We saw robotics as a high-potential field where a lot of the same technologies that we built for the smartphone space was directly applicable,” said Sweet. “Over the years we have done computer vision, machine learning, and certainly a big emphasis on low power processing. Any kind of high-bandwidth sensory processing we were doing on the phone in a small and low-powered form factor was directly applicable to robotics.”
As the ground robots started to turn into flying robots, the company decided to release the Snapdragon Flight drone platform that aims to provide drone developers with a platform that would help them develop innovative designs faster and at lower costs. “The integration that was needed to put all the processing power in a small package to fit inside a smartphone is directly applicable to drones where weight is such an important consideration,” said Sweet. “Now we have a small very high processing power, but in lightweight form factor reference platform for drones.”
On the developer side, when Silent Falcon’s Bowen decided to dedicate his career to software development, he never imagined it would take him into a brand new industry. “It is just inherently interesting the different types of applications and solutions you get to work in,” he said. “From a software point of view, it is not that there are many unique aspects of it other than just the application itself, it is just being able to work in an interesting field and be a part of this emerging industry.”
Moving over to the drone industry was a seamless transition. According to Bowen, the new types of programming skills he had to learn weren’t necessarily drone specific, but more for control systems and embedded software. “Ultimately a lot of the building blocks that make up software is fairly generic and it transfers well between different fields,” he said.
“You get to integrate a lot of different kinds of programming, for example embedded developing is quite different from application development which is somewhat different from mobile development, but you get to tie them all together.”
