As organizations have shifted to a mobile-first approach, they’ve found that the main challenge is testing for the sheer variety of devices out there. Also, tests need to be run on various telecom providers in different parts of the world. This infinite web of possible combinations has prompted a need for automation and framework consolidation in mobile testing.
Testing for Android devices is more challenging than iOS, according to Shawn Wallace, principal architect at Centric Consulting, a business consulting and technology solutions firm providing full-service management consulting services.
Android sits at the top at 73% market share, and iOS is second at 26%, a report by statcounter.com showed. But, what makes testing for Android complex is that there are just so many more types of active devices that use it with various sizes and form factors — not to mention the many different versions of the Android operating system still in use.
Because of this sheer number of devices, buying out all of the devices to manually test them by rote is no longer a feasible option for most organizations, explained Joachim Herschmann, a senior director analyst at Gartner.
Buying everything is no longer the option for testing mobile web applications that run on a browser, or natively on the phone. “Organizations no longer can afford to even, let’s say, buy the top 10 most relevant mobile devices. In the old days, if it’s an internal application for employees, they could then say, well, there are only two or three mobile devices that are admitted in our organization,” Herschmann said. “This will definitely not work if we’re talking about an application that’s rolled out to the public because you just don’t know what people are using.”
According to Centric’s Wallace, the drastic form factor changes are not the most difficult to test for; they can be the subtle changes between devices, such as a bezel taking up a little more space on one device than another.
Sometimes, these subtle changes are the ones that can have a drastic effect on the crucial functionalities of the app.
“The classic example is that some elements are partly off-screen like a button or something you could potentially still hit it, or maybe not. But the point is that that is not a good user experience, and in an extreme case, it could prevent you from actually interacting with it. I actually had this yesterday with an app I installed where one of their buttons would settle low at the bottom. But every time I tried to hit it with my finger, I basically hit the return on an Android device,” Herschmann said.
One way around this is to develop flexible layouts in apps, and cross-browser testing on laptops and computers can help avoid compatibility and sizing issues during the testing process, according to Artem Minaev, the co-founder of the digital startup platform FirstSiteGuide, a service that provides tools and tutorials for bloggers, webmasters, online entrepreneurs, and more.
Adding to the already crowded device market, IoT devices also factor into conversations around mobile testing. “There’s a reason why we see so much news about not only security issues, but simply just bugs, because there is very little thought given about what’s happening on that device,” Gartner’s Herschmann said.
Smart devices expanding
With the expansion of smart devices, mobile testing even encompasses testing your car as a mobile device.
“You could even consider the Tesla car itself as a mobile device because, at the end of the day, it’s at the end of this connectivity because it’s connected to the internet. You can set updates to your Tesla, so if you think about it that way, it becomes clear to ask yourself how are we testing the software on the Tesla,” Herschmann said.
Another major challenge with mobile app testing is how a mobile app renders while using specific telecom providers and whether that’s through 5G, 4G or 3G.
“I mean … if you have an iOS phone, how many times do you use Safari and it stops responding per week? It probably does it three or four times a week. Well, let’s do this at scale. Let’s run 10,000 tests or 5,000 tests. How many times is our app going to fail? Not because the app is broken too, because Safari just fails for whatever random reason,” Wallace said.
The sizable mobile app testing providers offer the ability to, for example, access a device in the U.K. through 4G, or access it through an Indian operator with 3G, or to access the same device in the United States through AT&T using 5G, Herschmann explained.
“So technically, three times I’m accessing exactly the same phone, but the difference is how I access it. And I bet you will see differences in the test results because latency is different; the apps may react differently to the kind of network connectivity with latency, jitter and that kind of stuff,” Herschmann said.
Another tricky aspect of testing is how an application behaves when switching between different providers, a situation prevalent in Europe due to its numerous providers in various countries.
Other things to consider are when signal strength goes down or if the user goes into a room with thick walls.
These scenarios are where crowdsourcing providers, in which people run software on their mobile phones to inspect for failures, come into play. Their input is then vital for regression testing, feature testing, usability testing, user story testing, cross-browser testing, accessibility testing, performance testing, routine sanity testing and more.
Solutions to device sprawl
While manual testing is necessary for certain aspects, the difficulty with testing manually is that the number of tests needed to test the codebases increases exponentially whenever a feature is added. “If you’re adding a new feature, you need to test all of the old parts as well, which can slow you down,” Centric’s Wallace explained.
“Two years ago, we were talking about ruthless automation, just automating everything. Now, it’s about let’s just automate what we need to automate,” said Diego Lo Giudice, vice president and principal analyst at Forrester. “You don’t want to automate something that just needs to run once. You want to automate things that are repetitive, but if you’re building Agile and DevOps, anything you build in sprint one becomes kind of a regression test in sprint two. And so if you don’t want to manually rerun those tests, you’re better off automating them.”
The goal is to minimize the number of tests that have to be run on the user interface on the mobile application and isolate some of the business logic into the APIs or server-side code where there are great patterns for testing at scale and testing in a continuous integration environment, according to Centric’s Wallace.
Giudice said that automation tooling for mobile has reached the same level of maturity as it has for browsers.
For browsers, the standard has been Selenium. Now, the standards that have grown around mobile testing include Appium, Cypress, and the ones maintained by Google and Apple for their applications on Android or iOS.
Still, highly skilled test automation engineers are required to “write code” for automation test scripts, which has opened the need for low-code tooling for testing and new advancements in AI.
The future of AI in mobile testing can help teams know which tests to run in a complex environment and do it for them; however, this technology is still in its early stages, according to Herschmann.
“The message I always give clients today is look, this AI stuff is not fully worked out, it’s the early days, but start looking at it today because you will take some time to understand and really comprehend it. And so you can’t just sit back for another year or two and wait until someone else has figured that out; by that time you’re lagging.” Herschmann said.
While organizations should try to automate as much testing as possible, there is still no way to get away from manual testing completely.
“I know of cases where literally, they have manual users that are in a car and they drive from Germany to the Netherlands, and they have to switch from their German network to their Dutch network as they’re driving and testing, how does my application react to this? So as you can imagine, that’s not something a developer could test locally on their machine. You would actually have to do something like this,” Herschmann said.
Another major trend around simplifying mobile testing is consolidating around a framework so that tests don’t need to be completely rewritten when dealing with new operating systems or devices.
It can get quite expensive when an organization has to make the decision between native development or using a platform such as React Native. Testing can be much more simplified when using one of the major frameworks by doing a lot of the testing before getting to the device because testing on the device is what’s painful, Centric’s Wallace explained.
Another way to do manual testing is by buying virtual machines, putting the emulator on them, and letting testers use a VM rather than using a device. This is effective for testing offshore with people in other countries in instances where they’re not allowed to install the app on their device or can’t have the data in their network. Still, they can connect to a VM because there’s no data transfer, Wallace explained.
“The paradigms of how I build an application have changed. There’s still native-app development, native in the sense that I write specifically with the widgets or elements that exist on iOS and those that exist on Android, but this becomes less frequent,” Gartner’s Herschmann said. “The more frequent approach is now to use one of the common frameworks that would take what you write in one abstract framework layer that gets sort of translated and compiled into the core platform that it’s actually rendered on.”
Also, the shift to using a public device cloud, which contains a couple of thousand devices for anyone to test on, is pervasive now. Whenever the next person wants to run their test, the devices are basically reset so that there’s no interference with any data.
To a much lesser extent, some organizations work within a private device cloud. Here the provisioning and administration mechanisms work the same way as in a public cloud, but the devices are still reserved for that organization.
The third and rarest option by far are the cases where there’s a device rack somewhere in a server room, but the way it’s accessed is also through the cloud, according to Gartner’s Herschmann.
“Over the years, it has shifted to the public cloud because the private option was a lot more popular a couple of years back when people didn’t trust the cloud so much,” Herschmann said. “Let’s just say if you’re a consumer application or maybe a retail website. Well, you want to put it out in the open anyhow. So there is no reason why you would want to do it in your closet and not go out.”
Cloud device providers often have contracts with various device and telecom providers to have day-one availability or even give developers access to the technology sometime before to run tests before launch day.
Simulators and emulators can also be used for more operating system features, such as Google’s new API for COVID, which isn’t very hardware hungry. On the other hand, for non-standard uses of the camera, LiDAR, or other features specific to the hardware, you have to test on the device itself.
Mobile testing is vital in a mobile-first world
Mobile app development has been integrated into mainstream development, which has increased the importance of diligent mobile testing. Now, the same testing scripts can basically be used for desktop or mobile, which has also prompted the need for consolidated testing.
“We’ve gone from a world where people thought we were going to be mobile-first or mobile-only 5 or 6 years ago, but it ended up being mobile-first because mobile-only is not a reality,” Forrester’s Lo Giudice said.
So mobile has to be integrated into a broader omnichannel strategy because these applications that run on mobile also have to run on a desktop or a tablet.
The progress to mobile-only was mostly stifled by the pandemic, since people worked in front of their computers at home. Before, companies tried to push as much as possible onto mobile phones.
“I think the assumption that everything would go onto a mobile phone was just wrong,” Lo Giudice said. “I mean, the web is still well and healthy, but what changed is that if I have to design an application, I will think that design for the mobile phone, for the smaller screens and then scale it eventually to the bigger ones, rather than five years ago, the apps were thought for a large screen. And then we had to squeeze them onto the mobile phones.”
This mobile-first approach has led organizations to hire out specialists that can test for omnichannel applications.
Organizations are no longer just looking to hire specialists that only write tests for the current iOS version, for example, but that write tests against a standard so that new versions don’t require them to change the test code beyond some compatibility issues, Centric’s Wallace explained.
“I think a lot about what the person that’s going to maintain this, what their experience is going to be like, and what the shelf life of these tests are going to be because you maintain an app for a much longer time than you spend building it,” Wallace said.
“When we don’t run tests, it means our clients test our software. Software is always tested. We just like to test it closer to when the code was written,” Wallace added.
When establishing a testing strategy, organizations first have to think about the tech stack that they’ll be testing for and who will be doing the testing, whether that’s the developer, the technical side or the business side. There are solutions out there now that can tailor to each type of tester, according to Gartner’s Herschmann.
“It’s no longer that hard to write a mobile app. Anybody can do it basically with no-code tools, but building the right app is all about building an app that is useful for potential consumers,” said Herschmann. “So you have to have a very compelling value proposition to even get people interested. And then don’t screw it by bringing up a choppy version of that app, because that will kill it immediately. I mean, that’s why I think testing quality is so important because honestly, you’ve only got one chance today.”
Popular open-source testing frameworks for mobile
This partial listing of popular open-source testing frameworks for mobile was taken from a compilation by SmartBear, which also included commercial products. SmartBear is a provider of tools for application performance monitoring (APM), software development, software testing, API testing and API management.
- Appium is an open-source test automation framework for use with native, hybrid and mobile web apps. It drives iOS, Android, and Windows apps using the WebDriver protocol. It supports cross-platform testing and code reuse, writing tests with multiple programming languages and offers many integrations.
- Espresso is a native open-source testing framework for Android to write UI tests. The tool is a part of the Android SDK and is easy to use for native mobile development. It offers automatic synchronization of UI elements and test actions.
- XCUITest is an open-source testing framework for iOS apps that is developed and maintained by Apple and is ideal for iOS-specific U.I. test automation. XCUITest enables developers or QA engineers to test the user interface of native iOS apps using the Swift or Objective C programming languages.
- Robot Framework is a generic open-source automation framework that can be used for test automation and robotic process automation (RPA). It has easy syntax, utilizing human-readable keywords. Its capabilities can be extended by libraries implemented with Python or Java. The framework was initially developed at Nokia Networks and open-sourced in 2008.
Real users for real testing results
While automation is a vital tool in many aspects of mobile testing, it has to be coupled with manual testing when it comes to applications that rely heavily on location, as in the case of Bloc, a social events app that rewards people for attending venues.
“We suggest venues that are local to someone and a lot of the functionality of the app relies on people being in the same area. This makes it extremely difficult to outsource the testing to a third party because their testers will be dotted around the world,” said Josh Wood, the founder and CEO of Bloc.
The company at first set up simulators and faked people’s location but it still wasn’t a comprehensive enough strategy. Then, Bloc had to find and onboard its own testers, which became difficult when trying to scale to thousands of users, according to Wood.
Now, the company relies on internal tests from real users who don’t necessarily know they’re testing the app.
“The best feedback and log files we’ve got are from real users using it in real time and that’s how we’ve created a pretty decent app today that works well,” Wood added.
