“DevOps is a powerful metaphor. We think there needs to be another one for mobile: DevTestOps, or DTO, as we’re calling it. Otherwise, you’ll be provisioning very quickly and agilely, but your customers will go away and you’ll never see them again,” said Tom Lounibos, CEO of SOASTA, a cloud testing company that, along with much of its competition, is moving nimbly into the mobile testing space as a diverse new market for app-testing tools begins to form.
The need for quality control as mobile apps proliferate is obvious, but developers aren’t getting the message yet. In his April 2013 keynote at the first-ever Xamarin user conference in Austin, CEO Nat Friedman said that a survey by his company found only 8% of developers were testing their apps prior to release.
Is that shocking figure an exaggeration? New York City-based senior software engineer Greg Shackles doesn’t think so. About half the developers at his mobile testing talk (also at the Xamarin conference) said they were testing, “but that was a self-selected audience of people interested in testing in the first place. I’d definitely agree with Xamarin’s saying 8% over 50%. One of the problems with testing mobile is that it’s a pretty difficult thing to test.”
‘This is not the Web’
There are many reasons why mobile apps are different from Web software. The first one? The way app stores (there are now more than 70, according to Mobyaffiliates’ directory) funnel approved executables to consumers’ tablets and smartphones.
Former Web developer Arlo Leach launched Set List Maker, a show-planning tool for musicians, for iOS after teaching himself Objective-C. He wholeheartedly believes that app marketplaces make it easier to sell software: “It’s a pretty unique opportunity. I built a lot of products on the Web and they just sat there, and I felt powerless to spread the word. I’ve done no promotion for these apps and they found an audience on their own in the App Store.”
The ease with which highly functional apps that meet a specific need can find success is exceeded, however, by the risk posed by poorly tested apps that garner bad ratings. “The problem has been that companies are putting mobile apps out with minimal testing, and that’s increasingly coming back to bite them. Failure is extremely visible and costly,” said IDC analyst Melinda Ballou. She plans to write a mobile-testing market report soon, but said the current market is “eclectic,” with major players just beginning to announce tools and strategies.
There are bigger risks than just being sunk by bad ratings. Buggy, slow or downright destructive apps can be hard to fix, once released. “This is not the Web,” said Shackles, who works at OLO Online Ordering. “I say that because I find it to be a good comparison point. I’m a Web developer on top of being a mobile developer. With the Web, if you push out a bug, you can fix that, and the next time the user hits that page, it’s fixed. With mobile apps, you have to go through the submission process. In some cases you can expedite it, but even then you still have to wait for users to update. There’s a very big barrier to getting emergency fixes out to users.”
Further, unlike cloud-based software, apps, be they native or hybrid or even HTML5-based, are dependent on device-specific form factors, browsers and operating systems. Google allows hardware manufacturers to customize the Android OS. The platforms themselves are changing constantly, and at a pace unheard of in the PC era, where OS upgrades happened every five years or so. (As an example, Apple released iOS 6 less than a year after iOS 5.)
Apps gain privileged access to APIs, storage and other resources, unlike Web applications, according to Localytics, a mobile analytics tool. Apps are designed less for passive information consumption, revolving primarily around actions and multi-tasking.
Finally, there’s the aesthetic question. As Xamarin’s Friedman told his user conference audience, enterprise apps are starting to “suck less” thanks to the consumerization of IT. There’s a newfound expectation that apps won’t just function, they’ll be beautiful and immersive.
#!
Don’t be greedy
Mobile app beauty can’t remain skin deep, either. A major concern, often foreign to Web developers, is energy consumption. Batteries get sucked dry thanks to programming errors such as:
• Automatically starting the app when the device powers on, or having it run in the background when not in use
• Running wild with energy-hogging functions such as GPS, camera, accelerometer and other device sensors
• Uploading user information and downloading ads
• Not allowing the device to go into sleep mode (wakelock and the no-sleep energy bug)
Data usage, privacy and security are also concerns that many ignore, at users’ peril. “We continuously see enterprises not completely aware and educated with respect to the type of vulnerabilities mobile apps have. They’re more focused on coding the app and pushing it over to production,” said Bala Venkat, chief marketing officer for Web-security firm Cenzic, who claimed 15% to 30% of its new customers are developing mobile apps.
Tablets and phones can’t be network hogs, either, both for economic reasons (not incurring data charges, for example) and because they depend on intermittent wi-fi and spotty cellular networks.
According to Rebecca Clinard, a mobile tester blogging for Neotys (a maker of mobile load and stress-testing tools), apps experience “delays in response times, which in turn affect the duration that ports or sockets are kept open—an environmental resource usage that is frequently seen with mobile applications. It’s because of this variable network connectivity that the user experience isn’t always an absolute known: The speed of the network depends on your device, your network plan (wi-fi, 3G, 4G, AT&T, etc.), your geographic location, network connectivity, the network usage vs. bandwidth, and so on.”
From the developer perspective, these constraints mean they should reduce network dependencies. Techniques include “decreasing embedded requests, using local storage on the device for caching static files, enabling transfer compression, avoiding redirects, minimizing data content size, reducing the number and length of cookies, removing lint from code (white spaces and comments), organizing the delivery for incremental rendering, aggregating requests, and using PUSH behaviors. Creating lighter-weight mobile applications allows the overall end-user experience to be less dependent on the device network vulnerabilities,” wrote Clinard.
Stay safe
The App Quality Alliance (AQuA) Best Practice Guidelines is a helpful compendium of critical practices. It recommended that developers “use HTTP 1.1 with pipelining. HTTP pipelining parallelizes HTTP requests within a TCP session. It’s perfect for improving user experience in high-latency environments, but it’s important to note that HTTP pipelining requires implementation on both client and server.”
Above all, the computers we carry in our pockets are communication devices. As such, they must be able to make and receive emergency calls, as well as handle other interruptions, such as sending or receiving SMS and or MMS messages. As AQuA noted: “The user must be able to accept an incoming phone call while the application is running. It should then be possible to resume from the same point in the application at the end of the call, or a logical restarting point.”
Oh, and don’t melt the phone. That’s the advice of mobile testing expert JeanAnn Harrison, who said that mobile devices in heavy usage (especially while being simultaneously charged) have temperature limits and shut off automatically before their components fuse into a molten mess.
The mobile testing tool stack
Once developers realize what’s at stake in this booming mobile app economy, they’re halfway home. The next step is to recognize that the device emulators that come with most development environments won’t cut it, as far as testing goes. (“No matter what you’re doing, simulator or emulator testing is not enough,” said Shackles.) A burgeoning array of tools is available to exercise apps at every level, and outsourcing the whole affair is always an option.
Consumer-oriented companies such as Google, Microsoft, HBO, Amazon and USA TODAY are already fans of uTest, for example, which in addition to automated tools for functional, usability, security, localization and load testing of mobile apps, boasts 80,000 professional testers from 190 countries.
Partnering with established testing leaders and development platforms is all the rage. Joining with market leader HP QuickTest Professional is Perfecto Mobile, for example. “As mobile applications become business-critical, enterprises need automated mobile-testing solutions that plug into their existing quality processes,” said Eran Yaniv, CEO at Perfecto Mobile. “It is our mission to address these challenges with MobileCloud for QTP, the first mobile test-automation solution fully integrated with HP QTP.”
Also worth mentioning is Silk for Mobile, launched last year, which records test scripts for iOS, BlackBerry, Android, Symbian and Windows Mobile devices, and which uses a visual tool for business analysts and testers alike.
A commonly used tool among mobile developers is TestFlight, whose tagline is “beta testing on the fly.” Integrated into Xamarin Studio and Visual Studio as well as Android tools, this free tool distributes builds to testers and distribution lists, as well as helps manage the feedback from manual and automated tests.
In the same vein, Google Play Developer Console now lets developers select a small test group from Google Groups and Google+ to alpha-test apps. Apps that pass muster can be escalated to beta through to launch.
#!
Testing for gestures
“Compilation is only the first unit test,” said Shackles. While most testers argue that a significant amount of mobile testing must always be done manually, there are a variety of tools out for user-interface and acceptance-test automation. With a history of .NET GUI testing, Ranorex has expanded its framework to cover Android, iOS and Windows Phone apps.
San Francisco-based LessPainful (recently acquired by cross-platform mobile development company Xamarin) offers tools similar to Selenium WebDriver but optimized for the “vastly different” mobile space. Its open-source calabash-android and calabash-ios are low-level automation and testing libraries for Android and iOS. These enable test code written in Cucumber to interact with native and hybrid apps, exercising end-user actions such as touches or gestures, making assertions and capturing screenshots on devices.
An interesting feature of Cucumber is that it uses natural language that non-developers can understand, like the following:
When I touch the “rate_it” button
Then I should see the rating panel
SOASTA TouchTest is another option for automating mobile application testing, and gestures will only become more important over time, according to Lounibos. “Every mobile app is built on gestures such as swiping to unlock the screen,” he said. “But we’ll start to go to more advanced gestures in the next few years, like hovering.” He noted that such gestures can be useful for surgeons and in other specialized settings.
But testing gestures requires a special approach, said Lounibos. “If you use an optical reader to capture the static view on the device, gestures can break the test. They become a pain in the butt,” he said. “Our engineering team started looking at it deeply and took a very creative approach. We embed a library into the object repository, testing inside the app. We capture all forms of movement, with precision. You can incorporate TouchTest into the daily build with Jenkins or Bamboo. Appcelerator uses it for their Titanium platform. Five-hundred thousand Titanium mobile developers are using TouchTest.”
The tool can automate more than 60 hours of manual testing in a single night, Lounibos claimed.
It’s the device, stupid
Fragmentation of the device market continues apace, and testing for every possible permutation bedevils even mobile development platform makers. Exadel vice president of engineering Dmitry Binunsky has been looking for a partner that could work with his company’s cloud-based app development tool, Appery.io.
“That is a little bit of a challenge, currently,” he said. “If you are on the cloud and not a device, we have a mobile emulator that can test all the features, except those on the actual phone. The challenge there is that all devices are different. There might be a card reader, or there’s no camera. There are QA companies that make a device list available on the Internet and charge you by the hour. We are talking to potential partners to simplify the process for our customers of testing on the device.”
Perfecto Mobile’s MobileCloud is one of an increasingly popular breed of service that essentially rents “a multitude of real mobile handsets and tablets connected to live mobile networks spread in different geo-locations,” according to the company’s marketing literature.
Another early investor in the mobile testing space is Keynote, which in 2011 acquired DeviceAnywhere and its customer base of around 1,000, according to an IDC report.
These companies have just been joined by Xamarin, which announced Test Cloud in April (based on technology acquired from LessPainful, now in beta). With a UI that reflects the company’s mobile focus, the test cloud, in conjunction with Calabash and Cucumber, will allow automated testing on real devices.
“We tell all our customers you’ve got to start running on real hardware from day one,” said Friedman at the Xamarin user conference. “The reason we say that now is because we’ve seen what happens if you don’t. You spend months developing your app in a simulator, you get ready to release, and you find out, ‘Oh, there’s a memory limit.’ ”
“I’m anxiously awaiting my invite to try it out,” said Xamarin fan Shackles. “I think it’s going to solve a huge problem. You’ve got a handful of iOS devices you need to test against, but on the Android side, there are thousands of different devices, screen sizes, and Samsung, Motorola, and the others all provide their own layer of modifications on top of Android.”
#!
Perform quickly, think globally
Once device concerns have been assuaged, it’s time to think about performance, advised Wayne Ariola, VP of strategy for Parasoft, which has a 25-year history in the code-quality and testing business. “Too often companies believe that functional testing alone is a replacement for truly understanding how it’s going to perform in the wild,” he said. “Since there is a massive rush to market, enterprises are not taking time to remediate performance. The first thing they need to be doing much better is designing mobile apps for specific geo-location.”
Parasoft’s service virtualization can help, Ariola claimed. Parasoft Virtualize allows developers to “simulate how things will perform given bandwidth, jitter, packet loss. Our service-virtualization environment can simulate all that chaos. No matter what mobile phone you have, when you push a button, it’s sending something somewhere to make a request, then waiting for a response. Your phone has [communication] protocols that it’s managing: SMS, telephone, JSON calls or REST-based calls. All that’s being sent over telco or wi-fi network for app to perform. Unless you have a game, anything transactional requires a connection to the outside world.”
Neotys NeoLoad is another mobile solution that gives you the ability to load-test both Web and native applications on mobile devices under real network conditions. SOASTA, too, offers this type of load testing as part of its bread-and-butter CloudTest platform. Testers can run browsers and real mobile UI testing simultaneously with protocol-level load testing. The company claims differentiation around its use of real users in addition to virtual users. SOASTA can also track browser-side metrics and on-device battery and memory performance.
Analyze this!
The last piece in the app puzzle is analytics, so that companies can learn from mistakes and customer behavior. Thanks to its analytics tool mPulse, “We’ve seen process compression. Developers are beginning to watch these screens… They want to understand why they’re seeing red. Everything is real time,” said SOASTA’s Lounibos.
Along with bandwidth and site metrics, and page or native app load time, mPulse tracks engagement via bounce, exit and conversion rates. mPulse also notes user location, device type, carrier speed and application usage.
San Francisco-based Crittercism claims to be the world’s first mobile application performance-management platform, with dashboards of real-time metrics as well as debugging breadcrumbs to help discover the causes of crashes or handled exceptions. Applause is another mobile app analytics service, launched this year by uTest. There’s also the aforementioned Localytics.
Another option is Flurry Analytics, launched in 2008 and boasting funnel analysis, custom segmentation and conversion tracking, audience demographic estimates, benchmark comparisons, and cross-application funnels. Google Analytics for Mobile Apps SDKs is also available for Android developers, of course.
Hands still needed
Ultimately, devices that are carried in pockets, touched, spoken to in myriad ways, pressed to ears, and—soon enough—worn continuously on the face or other parts of the body, will need more manual testing than traditional Web and desktop software. According to Harrison, manual is still best, not to mention most economical, for tests of trainability, configuration, performance and usability.
Even Shackles, a testing expert, doesn’t yet use any virtual mobile lab testing. Like many developers, he tests on a handful of devices manually. “It’s a relatively new space, and there are a lot of extremely hard problems to solve,” he said. “If you have a farm of devices to run tests in, even the infrastructure side is difficult. I try to put as much into unit-testable layers, because the more the surface area of your code is covered by automated unit tests, the more when you run on the device, you can focus on UI.”
It all comes back to DevTestOps, said Lounibos. “In mobile, you don’t have minutes to repair a problem. You have to do testing and some level of predictive analytics before you go live, or you risk not being in business very long. We all got caught up in the cool things we could do, and we forgot about quality. That’s not unusual for the beginning of a marketplace.”
#!
Hello, Fail
A leader in the mobile testing space, uTest has done an excellent job of highlighting what can go wrong on smartphones and tablets. “There’s no way to hide poor mobile app quality in the era of social,” the company quotes Michael Croghan, mobile solutions architect for USA TODAY (and uTest customer), in its e-book on the topic.
Indeed, pictures of app fails are legion on the Web. “Here’s a picture of what happened to me when I was in the airport trying to change my seat on Southwest,” said Parasoft’s Ariola. “If my mother got this message, she’d say, ‘What’s wrong with JSON?’ The middleware, which we simulate, returned an invalid JSON string. What happened here is it was a concurrency issue. Two services shot out from the app when I made a request. It was a race condition, essentially.”
Shackles cited the well-publicized example of Amazon’s Kindle for iOS update. “They released an update that, when you ran it, the user’s Kindle media on the device was wiped out,” he said. “The best thing they could do is update the release notes for the app in the app store. I know on Android there are ways to retract an update. That would partially solve the problem.”
Then there are regulatory questions that are sure to become more pressing over time. In May, the Food and Drug Administration warned a mobile app developer for neglecting to obtain 510(k) medical device clearance for the uChek Urine Analyzer app. The FDA argued that using the phone camera to read urine dipsticks meant “the phone and device as a whole functions as an automated strip reader” that required clearance as a “urinalysis test system.”
Finally, consumers are well aware that certain apps are battery-hungry. Purdue University researchers found that no-sleep energy bugs can suck the electrons out of inactive handsets in five hours. Popular offenders included Google Maps, Facebook and K-9 Mail.
Cross-platform: The third wave
Apple aimed for beautiful user interfaces. Android aimed for ubiquity. At his company’s user conference, Xamarin’s Friedman made the case for delighting developers as part of a third wave of mobile application technology. “Who was focusing on the developer? We decided that that’s what we were going to do. We were going to focus on delighting developers every step of the way toward their app,” he said.
C#-based Xamarin is the latest company to move into cross-platform app creation, joining Appcelerator’s JavaScript-based Titanium; FireMonkey’s Delphi approach; RubyMotion; and a Java-based approach from Oracle, among others.
So is there a need for better developer experiences like what Xamarin is promising? “In my opinion, absolutely,” said Greg Shackles, a mobile developer with OLO Online Ordering. “Historically, Android tools are not that good. They use the Eclipse IDE. Even the language—Java—hasn’t evolved in a long time. Then, for Objective-C, you have an Internet full of people who have problems picking up that language.
“Language-wise, Xamarin is right on top of the new shiny stuff. It’s leagues better than Objective-C and Java. You plug into Visual Studio, or use Xamarin Studio’s free IDE. It’s still maturing, but better than what’s out there.”
