There are a number of reasons why companies avoid automated UI testing for their mobile apps. Many are new to mobile development and are in a sort of Maslow’s mobile hierarchy of needs, in which perceived survival depends on just getting the app working, and things downstream in the workflow seem like luxuries. Even teams that have reached self-actualization with automated testing for Web development may be unfamiliar with the unique set of tools and processes mobile testing demands.
In fact, even the most sophisticated development organizations are often using only manual mobile testing for mobile—or not testing at all. According to a 2014 Xamarin survey, just 13.2% of mobile developers perform automated UI testing, and according to a Forrester Research survey, only 53% of developers perform even a cursory test on a single device.
Here are five of the most common reasons mobile teams have not yet automated mobile app quality, and five reasons why they just don’t make sense.
Myth 1: Speed. “We can’t take the time to automate.”
In 2014, vendors introduced nearly 7,000 new types of Android devices, as well as nearly 10,000 mobile-specific APIs. Mobile applications change quickly and ship faster. With QA in constant crunch mode, there just isn’t time to create test scripts and keep them in sync with features that change every day.
Reality: “You’re wasting time right now.” It’s true: Manual testing can be faster than automated testing, but only on the very first test run. On subsequent runs, any marginal benefits manual testing might have brought erodes almost immediately.
With every repeated test run or feature addition, app developers must either add testing resources or scale back test scope. With finite budgets, this will ultimately lead to a viscous cycle of diminishing quality. In response to negative user reviews and engagement data from untested devices, teams will want to expand device coverage. This will further increase stress on a QA department that is already at capacity, as the business struggles to research, procure and maintain devices while executing tests.
Even the best-funded manual UI testing programs fall short of complete coverage. In the U.S., mobile teams need to test on 188 devices to cover 100% of market share, yet according to a 2014 Xamarin survey, the majority of development teams frequently test on 25 or fewer devices, and more than a quarter of developers target five or fewer devices.
In real-world testing situations, automation can pay off almost immediately. On their very first test run, we have seen customers accelerate their testing timeline by 4x over entirely manual testing when running against 50 or more devices. Subsequent runs were several times faster yet, shortening nearly a full week of testing to just a few hours using.