If you’re like most organizations that develop mobile apps, you have some kind of systematic mobile software testing in place. You might even be using automation frameworks to execute your tests, and you might be testing across a high number of devices, browsers and operating system versions.
But if you think that makes you a standout organization when it comes to mobile testing, think again. A fully developed mobile testing strategy includes more than just simple test automation and broad device coverage.
To me, the best way to explain what goes into the most effective mobile testing routines is to think in terms of a mobile testing maturity model. This article discusses what I see as the four maturity stages of mobile testing, and how organizations can advance their maturity by leveraging techniques that go above and beyond the basics.
The stages of mobile software testing
The approach that most businesses take today to mobile testing falls into one of the following four maturity stages.
Stage 1: No mobile testing plan
Some businesses have no systematic mobile testing strategy in place at all. They perform testing on an ad hoc basis, if ever.
Fortunately, most organizations today have evolved beyond this stage because they realize that having some kind of testing regime is critical for identifying problems that can undercut the user experience in mobile apps. That said, organizations that have only recently begun to develop mobile apps, or that don’t frequently update their apps, may not have a systematic approach for testing them.
Stage 2: Manual testing and low device coverage
Businesses at a slightly higher maturity stage of mobile testing perform tests on a routine basis, but they rely heavily on manual tests. They aren’t automating tests, which means the tests are inefficient to repeat. They also struggle to test across a wide range of mobile device, operating system and browser types because they lack the resources to run tests manually on many different types of environments.
Stage 3: Automated testing and high device coverage
When organizations advance to automated mobile testing using frameworks like Appium, they’re also to run tests across a wider variety of mobile environment configurations. That leads to better test coverage and a lower risk that users will run into problems on their particular devices that the business didn’t test for.
However, just because you’re testing across a wide variety of environments doesn’t mean you’re testing everything you should test. You might be running tests for only a subset of all available application functionality, for example. Or you might be ignoring certain categories of testing, like accessibility testing. These considerations may be overlooked because your business lacks the time or resources necessary to implement automated tests for every test priority.
A second pitfall that some businesses sometimes encounter when they automate mobile testing, but not in a mature way, is that they struggle to interpret test results. When a test fails, they can’t quickly determine which change in their application triggered the failure. Nor can they efficiently pull out data from their tests in order to assess things like exactly how long a page inside an app took to load or exactly when a crash occurred.
Stage 4: Testing for everything, everywhere
Businesses that overcome these challenges advance to the highest level of mobile maturity testing. They gain the ability to test every aspect of application functionality, perform every relevant category of test and run tests for every environment configuration that their users might encounter.
On top of that, they draw on automations not just to run tests, but also to help interpret test results. Rather than manually examining failed tests or parsing files to identify the timing of different events, they collect relevant information automatically, which saves time and helps them to scale even further.
Evolving your mobile test maturity level
Another way of framing mobile maturity testing is to say that the more your testers are leveraging automation, the more advanced their testing strategy is.
But to be clear, I’m not talking here just about automated testing frameworks. Those only get you to the third maturity level.
Instead, businesses should take advantage of automation tools that can do things like generate test scripts, which makes it easier to support more test cases, or automatically repeat tests from on other devices in order to maximize coverage without requiring manual deployment of every test.
When you automate all aspects of mobile testing – not just the execution of the tests themselves, but also the process of creating tests, deploying tests and interpreting test results – your mobile testing strategy becomes as efficient, scalable and mature as it can be. That’s ultimately what separates the best from the rest when it comes to mobile testing, and it’s what every business should strive for if it wants to delight users while also keeping testing operations efficient.
