You’re sunk deep into your leather seat, enjoying the scenery that blurs by as the adaptive cruise control in your luxury sedan interprets data from the radar headway sensor and longitudinal controller to keep your car a safe distance from others on the road. This machine practically drives itself. Life is good. Your highway traffic-analysis app starts beeping, slowly at first, then more insistently. Nothing’s wrong, as far as you can see. Suddenly, maniacal laughter blasts from your stereo, and you instinctively cover your ears. You feel a horrible grinding vibration in the seat of your pants as your vehicle loses speed. “Damn you, Anonymous!” you yell, shaking your fist in the air.
It’s a little far-fetched perhaps, but the vulnerabilities of the systems embedded in your vehicles are not. The 2014 Infiniti Q50 was singled out by researchers Charlie Miller and Chris Valasek at the Black Hat USA convention in Las Vegas, along with the 2014 Jeep Cherokee and 2015 Cadillac Escalade, as “most likely to be hacked.” Radio, Bluetooth and telematic components used the same network as the engine and braking systems.
“If you’re allowing an app to interact with your car, you want that app to be tested rigorously. You don’t want to be a member of the extended QA team,” said Carlo Cadet, the lead technical evangelist at Perfecto Mobile. But nearly half of all app defects his company found in a survey (“Why Mobile Apps Fail”) are reported by users once the apps are in production. Twenty percent of these bug reports, not surprisingly, come via negative app store reviews, they found.
(Related: Threats that grow larger in mobile)
A major obstacle, according to Perfecto’s report, is device proliferation. Of the 900 mobile app practitioners surveyed, 63% were hard-pressed to test across the required number of devices and OS versions. Though the results aren’t exactly objective, given that Perfecto offers a cloud-based Device-as-a-Service testing tool, they ring true. In a similarly obvious vein, SmartBear Software, makers of the TestComplete tool for instrumenting and recording object-oriented smartphone tests, found nearly 50% of customers delete apps if they find a bug.
These modern versions of the classic Chaos Report (now debunked) are not deterring many from publishing half-baked mobile offerings in app stores, however. According to Genefa Murphy, director of mobile product management, analytics and user experience for HP Software, there has been an increase in interest in app monitoring. Unfortunately, she said, the motivation is often a Band-Aid approach to testing.
“In the need for speed, customers say, ‘Maybe I’ll negate testing and just put my app out there.’ When they do that, the sensible customers say, ‘I should be monitoring that app to see how many crashes are happening, on what device and OS, and in what phase of the end-to-end use case,’ ” said Murphy.
Enterprises looking to launch dozens or even hundreds of mobile apps are often aware of security, service-level agreements, and government or privacy compliance issues, and are more prone to choose a solution like HP Fortify to scan their apps. For the unwashed masses, however, test-driven mobile app development is an economical approach. “Because many of the hundreds of thousands of apps in the App Store are produced by micro-ISVs, anything that can improve the quality of an app without requiring much investment is a good thing,” writes Graham Lee in the book “Test-Driven iOS Development.”
Ever-shortening development cycle times, poor test coverage and device proliferation aren’t new problems; they’ve been gathering momentum for some time. Similarly, the push and pull between manual and automated testing goes back more than a decade—pre-dating the global love affair with the smartphone.
The persistent problems are the most human ones:
• Dealing with criminals, spies and other nefarious characters
• Detecting how people really behave rather than what they report
• Delighting users with mobile apps that meet their needs in surprising and useful ways.
The good news is, the mobile economy may finally be pushing the art of testing to the forefront in tackling these conundrums.
Mobile lacks security incentives
Whether or not there’s a crisis in mobile quality, there’s no denying, in a post-NSA PRISM world, that security and privacy vulnerabilities abound, and even the feds may be paying more attention to our personal activities than we’d like.
Ironically, many of these security holes aren’t new at all. “When we say mobile app, we increasingly mean a composite app that relies on a series of third parties for content and delivery,” said Cadet. Those third-party APIs are a major risk for mobile apps, the Open Web Application Security Project (OWASP) has found, and could be used to compromise data on the mobile device when transferred to the back end, or to attack the back end through the mobile application.
“Some of this stuff makes you want to bang your head against a wall,” said Ryan English, global director of HP Fortify on Demand. “They say, ‘Let’s go build a mobile app with these Web services.’ So you’re extending the surface area to talk to that app. Developers expect that the mobile apps are the only thing to worry about, so they forget to do authentication, authorization, transport layer protection, etc.”
In a white paper published November 2013, English’s team reported that, in a study of Fortify clients, 71% of vulnerabilities resided on the Web server. The paper blames mobile’s breakneck speed for the lack of care, noting, “We also see a resurgence of a lack of knowledge when it comes to Web service or API security, which we think is correlated to the use of frameworks or development shops that have no security incentives.”
Internet of Things opens (back) doors
The Internet of Things (IoT) represents a vast new landscape for data thieves. “We’re at a crisis point now with regard to the security of embedded systems, where computing is embedded into the hardware itself—as with the Internet of Things,” warned security expert Bruce Schneier. “These embedded computers are riddled with vulnerabilities, and there’s no good way to patch them.”
English’s HP Fortify team recently purchased and analyzed 10 IoT devices: a smart TV, webcam, home thermostat, remote power outlet, landscape sprinkler, device control hub, door lock, home alarm, scale, and garage door opener. The results were chilling.
“Network traffic is not encrypted. You can gain root access to these devices and do things like shut down refrigerators,” said English. Of course, spoiled food is the least of his concerns. Privacy was minimal, with devices collecting sensitive information such as name, date of birth, credit card numbers and the like. Data flew unencrypted on the home network, and remote control of often-insecure firmware is the name of the game.
Though IoT represents a tiny fraction of the use cases test automation companies such as HP, Perfecto, SOASTA and others are currently encountering, there are sensors everywhere, along with new ways to spoof them. “Mobile is dropping in new features very fast, and enterprises are struggling to respond,” said Cadet. “We’re tackling fingerprinting right now. Sensors in general are multiplying. Samsung is dropping a lot of sensors in those devices.”
Samsung’s recent purchase of Kickstarter-launched home-automation company SmartThings means many more sensor-driven features to come.
Meanwhile, it seems Samsung learned little or nothing from the iPhone 5 fingerprint recognition debacle: The fingerprint lock in the Samsung Galaxy 5 was easily defeated by researchers who then gained access to PayPal or banking accounts, which, wisely or not, can be associated with the biometric authenticator. The spoofed fingerprint could be photographed on the smartphone screen and recreated with wood glue. This dovetails with findings by University of Pennsylvania researchers who found that “smudge attacks based on reflective properties of oily residues are but one possible attack vector on touch screens” and concluded that “the Android password pattern, in particular, should be strengthened.”
Mobile payment systems have yet to proliferate in the United States, but once they do, experts worry they will be primary targets for fraudsters and skimmers. Google Wallet uses Near Field Communication (NFC) to transmit actual card data through the mobile device for proximity payments, while Apple devices don’t contain NFC transmitters. Instead, they use a less-risky “non-NFC” payment solution that doesn’t keep sensitive authentication data on the device. Neither approach has been sufficiently tested in the real world.
Finally, the development platforms themselves must be safe. “Write once, multiply out is the promise of hybrid or HTML5 apps,” said English. “Like Appcelerator…we’ve never tested their stuff. But if Appcelerator had a vulnerability in how they write an Android app, that would be an exponential problem. That’s something we’ve got to research, because no doubt they’re here to stay.”
More than Mini-Me
“The siren song of one-design-fits-all-screen-sizes has a long history of bright hopes, broken promises, and weary designers and developers,” writes Steve Krug in “Don’t Make Me Think, Revisited,” the 2014 version of his now-classic tome on Web usability, first written in 2000. “If there are two things I can tell you about scalable design (a.k.a. dynamic layout, fluid design, adaptive design, and responsive design), they’re these:
• It tends to be a lot of work.
• It’s very hard to do it well.”
Krug’s mobile-specific advice isn’t too surprising, or extensive. In his chapter on mobile (entitled “Mobile: It’s not just a city in Alabama anymore,”) he notes that capacitive touch-screens can’t detect a finger hovering over a menu item. This is a major departure from Web design. “As a result, many useful interface features that depended on hover are no longer available, like tool tips, buttons that change shape or color to indicate that they’re clickable, and menus that drop down to reveal their contents without forcing you to make a choice. As a designer, you need to be aware that these elements don’t exist for mobile users and try to find ways to replace them.”
He also rejects the flat interface design trend, which removes some of the textures previously used to convey information, such as shading on buttons and font changes. “By removing a number of these distinctions from the design palette, flat design makes it harder to differentiate things. Flat design has sucked the air out of the room.”
User experience: more important than ever
Design trends come about for many reasons, including the whims of designers, which, while prettier, aren’t necessarily any better than the whims of developers. Hence the need for user experience experts and UX testing.
“People are still not hiring user experience experts,” lamented HP’s Murphy. “A lot of companies make the mistake of leaving design and UX to developers.” She points to GE Software as a vocal proponent of design programs.
Since Thomas Edison founded it in 1890, General Electric has made many things. But software is a relatively new development—and user experience an even more recent focus, according to Greg Petroff, general manager of GE’s UX Center of Excellence.
“GE is a very strong engineering-driven culture. They have incredible stuff that they’re making up, but they usually don’t know what to use it for,” he told attendees at the 2012 Managing Experience conference. “That’s probably a common refrain when you work with engineers. We believe if we can bring design as a cultural strand of DNA to the company, we can unlock a lot of the innovation that’s growing inside the company.”
Unlocking app utility is the purview of Tomer Sharon, a Google search user experience researcher. In his talk at the 2014 Google I/O conference, he explained that users lie. The trick is to ask the right questions.
“A great user experience sampling question asks about repeated behavior: something that happens a lot during a day or study period,” he said. Bad questions, according to Sharon, can be answered yes or no, are quantitative, or request an opinion.
But observation is even more important than asking questions, the noted Web usability consultant Jakob Nielsen maintains. In a 2002 article on the trend of using anthropological methods to study customers, he pointed out that “The reported studies emphasized interview questions, even though quietly observing users is more valuable and the real reason to go into the field.”
Understanding device context
“If there was a word for the last year in mobile testing, it’s ‘contextual,’ ” said Tom Lounibos, CEO of SOASTA, a cloud-based performance testing company. “Get more context for user experience so you can make better recommendations, give better deals, understand buyer needs by what’s in their shopping cart. They used to call it user experience, then in retail they called it consumer experience. Now they call it contextual.
“It’s all about the app. The right app makes technology bloom; the wrong app makes people look like ‘glassholes.’ If you’re running around taking pictures of people, that’s pretty creepy. But if when I get off the plane, an attendant wearing Google Glass can tell me my connection, that’s useful.”
As a company that uses real devices so that testers can evaluate apps on a representative spectrum of platforms, Perfecto has to watch the market carefully, Cadet said. “Every time a brand new sensor is released to the market, we try to map the sensor to the use case. We try to understand the top use cases—image capture, voice capture—and help the organization shift from manual test to automated test.”
Not every sensor makes the cut, however. “Theoretically, there’s a way to watch eye movement. We don’t think that’s a killer feature,” Cadet said. Presumably, it joins humidity detection as a less-useful sensor. It remains to be seen whether infrared sensors for night vision and surveillance will take off.
Of course, those sensors can go haywire. It took BlackBerry engineers about a day to discover that a server was sending corrupt sensor calibration data to phones, causing temporary home screen freezes, proximity sensor fails, screen rotation fails, and errors in the compass app. Once the culprit was found on the server-side, the solution was simple for end users: Restart the phones. Considering the fact that there are five to 10 sensors on a phone, it’s promising that sensor-related bugs aren’t more common. One thing is certain: The complexity of smartphones and connected devices will only grow.
Mobile testing matures
While the Wild West days of mobile app development are coming to a close, using tools to plan and execute testing earlier in the development cycle is still aspirational for most organizations, according to HP’s Murphy: “If you’re going to do mobile device testing, automation is key. If you don’t have fundamentals like automation, I would spend money on automation and outsource management of devices to a third party, then once you have that in place, you can build a device lab,” she said.
Murphy observes that 60% of her customers are starting to do continuous integration: “Mobile teams tend to be more self-managing, so they can implement that slightly easier.”
Perfecto’s Cadet says the opposite: “Many are struggling to get continuous integration to work for mobile.” He does note, however, a shift from “loosey-goosey” projects to formal mobile programs applying appropriate practices and test automation—all under the pressure to launch quickly and brilliantly: “It’s not the or of velocity and quality, it’s the and. Now organizations are struggling to figure out the how of an increasing test coverage footprint.”
Ten ways to test your mobile app
1. Observe user experience in context, with or without technology. Paper prototypes can be fun. “Making your app delightful is a fine objective. Just don’t focus so much attention on it that you forget to make it usable, too,” writes Steve Krug in “Don’t Make Me Think, Revisited.”
2. Mock up and test prototypes with tools like Axure (for creating interactive wireframes with conditional actions) or Balsamiq (for stylish-looking, static drawings).
3. Use tests to drive development. “Test-driven development encourages building applications from the outside in. You know that the user needs to do a certain task, so you write a test that asserts this task can be done. That requires getting some data from a network service, so you write a test that asserts the data can be fetched. That requires use of a URL request, so you write a test for that use of a URL request. When you implement the code that passes the test, you need to code only for the use that you’ve identified. There’s no generic handler class, because there’s no demand for it,” writes Lee in “Test-Driven iOS Development.”
4. Find the bugs in your logic layer. “Most bugs don’t care what device you’re running on. If you can replicate OS version, screen size and memory constraints, only a few classes of bugs will slip through your net,” Thomas Knych, staff software engineer in test at Google, told attendees in his talk at the 2013 Google Test Automation Conference.
5. Use static and binary code analyzers to test for basic security principles such as safe string functions to avoid buffer and integer overflow. Better yet, avoid entire classes of unsafe code with programming languages such as Apple’s Swift, which is more productive (read: easier) than Objective-C, initializes variables before use, checks arrays and integers for overflow, and manages memory. Use a security auditing SaaS such as HP Fortify on Demand to find vulnerabilities in compiled code.
6. Test concurrent code for resource contention or scheduling problems in a variety of real-world environments.
7. Check APIs for vulnerabilities and bugs. Remember the security principle of least privilege and disable any unnecessary access granted by APIs.
8. Automate functional tests for iOS and Android apps with tools like Appium, SeeTest or MonkeyTalk.
9. Virtualize the end-user experience with performance testing using tools such as Neotys. “Remember, it’s not just the impact of load and scale. You’re also looking at network, carrier and noise,” said Murphy.
10. Test on real devices, not just emulators, for different screen resolutions and pixel densities as well as form factors and gestures. Use a Device-as-a-Service cloud such as Mobile Labs’ deviceConnect, Perfecto Mobile or SOASTA TouchTest. Remember that speed is more important than responsive design: “Be careful that your responsive design solutions aren’t loading up pages with huge amounts of code and images that are larger than necessary for the user’s screen,” writes Krug.
App security in the Internet of Everything
Founded in 2001, the Open Web Application Security Project recently partnered with the European Network and Information Security Agency on the Smartphone Secure Development Guidelines. Here are some of its key recommendations:
• Design for security, classifying data sensitivity and storage. Validate API calls to sensitive data.
• Handle passwords with care, or store semi-permanent authorization tokens on the device (OAuth model). Encrypt tokens in transit with SSL/TLS. Tokens can be issued by the back-end service after verifying.
• Protect data in transit.
• Turn on binary protections to prevent reverse-engineering, detect jailbreaking, reduce information leakage and more.
• Authenticate, authorize and manage sessions correctly.
• Choose session identifiers that can’t be predicted from a given random number generator seed, such as the date and time. “The standard method of using the date and time is not secure. It can be improved, for example using a combination of the date and time, the phone temperature sensor and the current x, y and z magnetic fields.”
• Secure back-end APIs (services) and the platform (server).
• Be careful about mobile app distribution and provisioning.
• Prevent unauthorized access to purchased resources such as wallet, texts, roaming data and phone calls. Check for anomalous usage. Warn the user of any costs about to be incurred.
• Ensure secure distribution/provisioning of mobile applications, bearing in mind that app stores delay security patch deployment but can also remove insecure apps.
• Beware of untrusted parties sending unverified input (extra levels in a game, scripts, interpreted SMS headers) that is interpreted as code. Injection attacks are the mechanism behind much surveillance, spyware and dialerware.
