What user and market data tell us about the challenges of test coverage today

I am frequently asked this question: “On what devices and browsers should I be testing my mobile and web applications?”

It’s a very valid question, because the volume of permutations is vast, and it gets bigger and more fragmented all the time. In 2020 alone we are seeing several major OS platforms debut. These introduce both new features and new challenges.

That’s why we developed the Mobile and Web Test Coverage Index, a free report to help organizations get deeper insight into their target markets and then create test coverage strategies based on known priorities. The results are based on our own testing on over 10,000 different devices and available market data across North America and the main markets in Europe, India, and Australia.

It is also important to understand the term “test coverage” and put it in the right context. Test coverage is a combination of covering both the right platforms that the end users would care mostly about together with high-value test scenarios that represent the most important application business flows.

The mobile market today

In the Android world, Android 9 is the OS with the biggest market share, at 39%, followed by Android 8 at 19%, and then the rest is evenly split across Android 6, 7, 10, and older legacy versions. These stock versions may then be customized by large device manufacturers, who add their own UI layers, apps, and proprietary functionality.

Using the example of Samsung (which takes the first four places in the top ten Android devices right now), its three generations of S7, S8, and S20 each run on different stock OS, as well as custom OS versions. So, that means there are many variations to include in testing strategies. Also, newer Android stock OS versions like the upcoming Android 11 are only supported for the near future on Google Pixel devices, forcing the Pixel to be a dominant device in every test lab.

Over in the iOS market, there is a monthly release cadence that fixes most bugs and performance issues. Apple controls market adoption and termination of older OS support more than Android, so being able to provide test coverage across the latest versions of iOS 13, iOS “latest-1,” and iOS 12 latest (for regression defects) is sufficient in most cases, though companies might also have to test iOS 11/10. The vast majority of devices introduced in the last four years today use iOS 13, and the same applies to iPads, with almost 80 per cent using iPadOS.

Regional variations may need to be factored in to decisions about what to test, as the data around the most popular mobile devices shows. In the US, the top three are the Apple iPhone X, iPhone 11, and Samsung Galaxy S10+. The two top spots are the same in Canada, but the Galaxy Note 9 comes in at number three. Over in Europe, the Apple iPhone 8 currently topples the iPhone X off the top place, with the Samsung Galaxy S9+ in third place.

Web

When it comes to the web, the situation is also becoming more complicated. Mozilla Firefox is releasing a GA version monthly. Google Chromium is now the basis for the Microsoft Edge Insider browser, and MacOS platforms with the varying Safari permutations are also imposing more test requirements. Each browser comes with a different rendering engine, which creates the website’s look and feel. For example, Chrome uses Blink, Firefox uses Gecko, and Safari is based on WebKit. 

Chrome is by far and away the most popular desktop browser at almost 70 per cent, followed by Firefox at 9.54 per cent, and Safari at 8.51 per cent. The dominant desktop OS is still Windows with over 77 per cent of users, followed by OS X at just under 18 per cent.  While each of the browsers has its own release cadence, they are also supported across different desktop OS versions, like MacOS and Windows, making the test matrix complex. In addition, the adoption trends across these browsers are completely different. Chrome browsers typically get adopted almost within days from the release, while Firefox browsers take longer for users to adopt.

Coming up

Google has already launched its first Android 11 beta to developers, mainly for Google Pixel real devices and emulators. Take advantage of betas rather than waiting for GA to ensure test automation scripts run properly and that there are not any regressions. Betas also help teams prepare the app for new features.

• In Android 11, new features include: 5G APIs, neural network APIs, and privacy changes (such as one-time permissions), so there is a lot for development teams to cover. Android 11 is slated for September-October GA. In September, there will be the usual annual slew of news from Apple, including the launch of iOS 14. The iPhone 12 series will be a few weeks later, most likely October. This means existing iPhones/iPads will have to be tested in parallel on both iOS 13 and iOS 14 before the availability of the next generation of devices with built-in iOS 14, which is already causing regression issues around memory consumption, UI issues, and more.

There will also be a new Apple watch series, new iPads, and as in any new version, performance improvements and bug fixes, together with some innovative new features. For instance, ‘App Clips,’ which — while exciting for consumers – will require more focus from developers and testers.

Best practices for test coverage

As the results of the mobile and web index show, the volume of releases, updates, and possible variations means that there is a huge amount to consider when designing test coverage strategies. The starting point is understanding the landscape needing to be addressed, and obviously the top-selling models in the market have to be the priorities. While the traditional approach was limited to what devices to test, these days it is essential to include other considerations. For instance, there are often variations within a platform: some devices might support facial recognition while others do not; and increasingly, there may be multiple screen sizes and resolutions. 

Here’s an example of a digital platform coverage strategy developed by one of our customers across unit testing, build acceptance testing, acceptance testing, and production.

• Essential coverage — Top 10 “must test” devices based on usage. 

• Enhanced coverage — Top 25 devices, including legacy and trending devices, and different screen sizes.

• Extended coverage — Top 32 devices, including niche, legacy, and brand new devices to represent the long tail. 

This strategy gives the company a balanced view of the bigger picture, so that it can map out test coverage more efficiently. 

As resources make it impossible to test everything at maximum frequency and scope, prioritize fiercely and use data to decide what to test, not intuition. We see teams prioritize based on instincts and personal judgement, but it is better to use data to remove the subjectivity and focus what matters most, namely delivering value to the customer. Also, the size of the test suite is less important than user-value. Focusing on the top 20 per cent of most important user journeys is more important than aiming for a nominal target such as 80 per cent test coverage. 

Mimic the real user

Real user conditions are essential to test coverage. Consider:

• Geographies — App behavior on an ideal “happy path” scenario versus being up against changing network conditions.
• Apps running in the background — Interfering with each other, exhausting device resources, and impacting app performance.
• Load conditions on both client and server side — App behavior when the backend is being loaded, in poor network conditions, or on legacy devices and OS versions.
• Localization requirements — App performance in various locations, different languages, or if translations are an issue on specific devices and screen sizes.
• Device system interruptions — Conflicts caused by incoming calls, system popups, and system resources will interfere with an app’s performance.
• Network conditions — e.g. 5G, 4G, good, average or poor; vertical orientation, or both vertical and horizontal orientation.

Test with frequency

Change happens fast, so it is important to be able to test over and over again, but if tests take five days to run, or use up a lot of capital resources, then that becomes unmanageable. So, shift left, integrate with the CI/CD, and get feedback to developers quicker. Automating tests as much as possible will clearly help, but balance that with smart analytics. This helps teams sift through all the noise created by the results so they can truly understand what is having an impact on the business.

And that brings us back to where we started: the importance of data in designing a test coverage strategy. As the saying goes, knowledge is power, so the more organizations can understand their markets and their users, the better the chances of developing a relevant test coverage strategy.