Being a software tester is no easy feat, especially when technology is constantly changing. In the past, when testers had to assess an application, they didn’t have to worry a whole lot about different operating systems, devices or screen sizes, according to Joe Schulz, director of mobile testing at Orasi.
“We have lots of different devices and form factors to have to test for now, whereas before in a Web-type application we just had to worry about TCP/IP as the protocol and Windows as the operating system,” he said. “We didn’t worry a whole lot about screen formatting, and we have to do all of that of course in mobile for the phones.”
Today, mobile application testers not only have to worry about these different devices, operating systems and form factors, but they also have to keep up with the number of devices and updates being released and whether or not their application is going to work when a new device or version of an OS comes out.
“If you take the last 60 days as an example, Apple has dropped a new release of iOS; Android is out with Lollipop; and in between them we have six or seven brand new devices,” said Don Addington, CEO of Mobile Labs. “We have to go back and see if the stuff we wrote last year and the year before works on these new environments and these new devices.”
Mobile testing strategies
In this new mobile world with the pressures to develop for multiple form factors faster and get to market first, software testers are left with little time and a lot of work. But with some guidance and recommendations, testers can keep their heads above the water from the flood of devices.
Test on real devices: Testers will often try to use an emulator first because it is free and easy to download, according to Schulz. But since emulator testing is done on the desktop, there isn’t any assurance that it will work on the physical hardware a customer is using. “Moving to production without testing on real devices at some point is probably the biggest mistake I see people make,” said Schulz.
Test on multiple devices: “You want to test on as many devices as you think your constituency is going to be using,” said Addington. With the variety of mobile phones comes different screen sizes for those phones, and testers should cover all their bases by ensuring their application looks good on all the devices no matter the model or size.
According to Schulz, there is no guarantee that functionality and quality are going to work across all devices unless they are tested for. “They have to be looking at a couple of dozen devices to make sure they are covering at least the 80% of their customers’ environments,” he said.
Automate: There isn’t enough time in the development life cycle anymore for developers to manually test everything, according to Schulz. “A developer has to unit test on at least five or six different platforms, and in order to do that they would have to run these unit tests five or six times,” he said. “They couldn’t possibly be able to do that manually. They wouldn’t be able to get their development work done.”
Addington added that automation is key when it comes to testing all the different instances of an application. “The price for putting out apps with poor quality is not good for the brand,” he said. “You can be deleted in a matter of minutes by the push of a button if it doesn’t work well.”
Manual test: Not all tests can be automated simply because automation can’t catch up to all the new functions being released every day, according to Schulz. “The automation vendors try to automate everything that you could possibility do on a mobile phone, but of course Apple, Motorola and Samsung are creating new techniques every day,” he said. “Manual testing has to fill that gap until automation can catch up.”
Those advanced use cases include activities such as being able to shake the phone, finger gestures and biometrics. “You have different kind of activities that you can do with mobile devices that you just can’t do with other kinds of devices, and because of that you are going to have to continue to test them in a manual fashion,” said Addington.
Test all versions of the operating system: According to Schulz, a common mistake testers make is only testing the latest release of an operating system. But in the real world, the very latest version probably has the lowest adoption rate.
Keep usability in mind: “In the past, we’ve been concerned for usability, but we assumed certain basic access through for example a keyboard for a mouse,” Schulz said.
“Usability wasn’t quite as important, people would wait a little bit longer in terms of performance, and so we focused more on the functionality. Nowadays applications are more oriented to function, so whereas a website is geared to an entire portfolio of functionality, a mobile app really should be focused on an individual task or a set of related tasks.”
Don’t wait for the app to come down the pipeline: Previously, development would create an application and give it to operations, and operations would not publish it until the QA team gave the thumbs up. Today, development teams don’t release to production. They release to the app store, and in the process, QA is often forgotten, according to Schulz.
“QA has to be more proactive about talking to the development teams and has to make sure they know what development apps are being written and insert themselves into the process actively as opposed to waiting for that app to come to them,” he said.
What’s happening in mobile?
With all the latest technologies and functionalities happening in the mobile device domain, SD Times spoke with some experts to find out some biggest trends happening in mobile and what testers can expect.
Schulz: “I think the biggest trends probably are just the proliferation of the devices themselves. Everyone has a smartphone now… We used to wonder what percentage of users had a smartphone, but everyone does for the most part. The biggest trend now is to take those apps that we are building and turn them from brochureware into active revenue-generating-type transactions.
“From a business perspective, the business is looking at ways to turn that into customer-facing revenue-generating kinds of opportunities. Very often that involves location. Location is probably one of the biggest trends in development these days. Businesses are trying to make an application location-aware and introducing some sort of predictive marketing. For example, trying to recommend certain stores based on where you are located and asked on what store you are in.”
Addington: “The whole world is going mobile. Some percentage of every corporation’s portfolio today has been mobilized, and if you look at whatever body of work is being done today, tomorrow that body of work…is increasing across these organizations. In order for testers to keep up, it is a matter of figuring out how to be as efficient as possible with our resources, and cutting out those steps that waste time.
“A lot of times it is just something simple, like you finally get access to the device, and the darn thing is not powered up, so you lose 20-30 minutes waiting to get access to it. It is really this chaotic environment that we are all placed in today at the degree to which we can bring some sanity to it. [Bringing] some organizational structure to it, like a private device cloud for example, is just key to be able to make people completely efficient and not to waste time on these monadic aspects.
A guide to mobile testing tools
Applause: Applause provides in-the-wild testing services that go beyond the traditional QA lab to create a great experience for app users where they live, work and play. With functional, usability, localization, load and security testing, Applause services span the entire app life cycle. Through a global community of more than 150,000 professional testers, Applause empowers companies to understand how apps will work in the real world, with real people on real devices, across locations and use cases that match actual users.
BlazeMeter: BlazeMeter provides performance and API testing for native mobile apps and websites. The company’s unique mobile app technology allows users to easily create tests by recording and playing back activity from real devices. The platform can simulate traffic from any device using realistic mobile network characteristics. BlazeMeter integrates with any CI or CD tool via plug-ins or APIs, and it can create load at any scale from multiple geographic locations.
Borland, a Micro Focus company: Borland has a variety of test tools that are open, agile and work across the entire application development life cycle. Its Silk Portfolio of testing solutions reduces cross-browser testing time, automates manual test processes, ensures peak performance, and streamlines testing across desktop, Web and mobile platforms. It covers automating functional tests, performance testing in the cloud, application performance testing, collaborative test management, cross-browser testing tools, quality assurance, and mobile testing.
HP: HP provides a suite of software testing solutions that enable organizations to address the needs of modern application delivery, from agile test tools that integrate into the developer ecosystem, to manual and automated functional/load testing that offer coverage for a broad range of technologies, from ERP to Web and mobile. HP’s testing offerings also provide virtualization capabilities that allow customers to simulate service and network conditions, facilitating faster release cycles and application optimization.
Keynote: Keynote’s Mobile Testing Platform is a cloud-based SaaS solution that provides manual and automated testing of all mobile apps and websites to ensure quality, high-performing mobile assets. With Keynote Mobile Testing, QA teams can quickly create and automate test scripts to capture, verify and replay real user interactions on live mobile devices. The solution combines the convenience and efficiency of test automation with the accuracy of live mobile interactions to generate reliable, reportable and actionable results.
Mobile Labs: Mobile Labs offers a private mobile device cloud that can drive up productivity and fuel enterprise growth. From manual and automated mobile website and app testing, to collaboration through instant access to real devices, deviceConnect brings order to the chaos of mobile development, DevOps, and testing. Mobile Labs solutions help conquer mobility obstacles such as multiple device types, agile pacing, constant operating system upgrades, and multiple operating systems.
Neotys: Neotys is a leading innovator in load testing and performance monitoring solutions for Web and mobile applications. Neotys’ products NeoLoad and NeoSense enable development, QA and IT operations teams to quickly and efficiently test and monitor the quality, reliability and performance of their applications. More than 1,500 organizations globally have selected Neotys’ solutions because they are agile, easy to use, and support all RIA and mobile technologies.
Orasi: Orasi Mobile Testing is focused on providing customers with a comprehensive mobile testing solution designed to fit their needs, from physical devices to simulators and emulators. Depending on the organization’s functional and security requirements, Orasi can provide public, private, on-premise or hybrid cloud solutions for automated mobile testing. In addition, Orasi’s testing services include usability testing, security testing, real device testing, performance testing, functional testing, network testing and emulation, mobile app test automation, localization testing, and device compatibility testing.
Perfecto Mobile: Perfecto Mobile’s Continuous Quality Lab accelerates mobile app development velocity without compromising quality. The Continuous Quality Lab is a cloud-based infrastructure that enables mobile app testing on real devices throughout all stages of the mobile app SDLC. With the cloud-based device lab available on demand, patented automated testing capabilities, and an open system that integrates into the tools dev/test teams are already using, the Continuous Quality Lab is the solution for delivering better apps faster.
SOASTA: SOASTA’s cloud-based integrated testing platform allows for smarter test automation and performance testing for the next generation of Web and mobile applications. The SOASTA platform provides accelerated and automated testing at scale, load testing, real-time analytics, visual test creation, cross-cloud-grid provisioning, and the ability to start, stop, pause and restart tests. Users can test on actual mobile devices without optical recognition, emulation, jailbreaking or tethering. Products include CloudTest for building, executing and analyzing performance tests; TouchTest for mobile functional test automation; and mPulse for real-user monitoring.
Telerik: Telerik Test Studio and Telerik Mobile Testing are sets of test automation tools that help teams be more efficient in regression testing, improving test coverage and reducing the number of bugs that slip into production. They offer automation solutions for GUI, load and performance testing for desktop, Web, hybrid and native apps running on any device (Android, iOS and Windows Phone). Tests can be authored in C#, VB.NET or JavaScript.
TestingWhiz: TestingWhiz is an agile Web test automation solution that enables mobile test execution for extending the capabilities on different mobile platforms. Through its intelligent Web recorder, it supports the execution of test scripts over Wi-Fi on a mobile browser. It also supports execution on all Android phones with support for native browsers. The feature is accessible for all TestingWhiz Enterprise users, and can be achieved within moments.
TestPlant: eggPlant Mobile is the combination of eggPlant Functional and eggOn, the mobile agent that enables eggPlant Functional to fully test mobile apps on iOS, Android, Windows Phone, BlackBerry, and other mobile devices. eggPlant’s unique image-based approach is the natural way to test user-experience-focused mobile applications, and it massively reduces the effort of cross-device and cross-platform mobile software testing.
Xamarin: Xamarin Test Cloud allows mobile test teams to automate tests in any language on more than 1,000 real devices through the entire development process. Testers can set up tests for different device platforms, operating system versions, form factors, and hardware capabilities. In addition, the test cloud provides detailed reports with results, screenshots and performance metrics.
