Testing in a modern, mobile and agile world

Test early and test often is a mantra every tester these days is familiar with, but with the advent of mobile and methodologies such as DevOps and agile, testing can get lost.

“Organizations are expected to put code out faster, and there are pressures to do that, said Kyle Cochran, vice president of product management at QASymphony. “But without the right tools and without the right methodologies, testing is a component that will get squeezed, and it will force organizations that have pressure to put out code faster to put out code with potentially more defects in it.”

Agile has made software development faster, causing testing to be more automated. And mobile has brought a new dimension to testing, forcing even Web apps to take into account mobile users. Today’s testers need to ensure functionality and performance on a wide variety of devices and network conditions to guarantee good end user experience, according to Jonathan Bracken, vice president of product marketing at Neotys.

So, how should testers implement testing in this new modern, mobile and agile world?

Testers need to remember that there isn’t one way of doing things, according to Cochran. “Testing requires a good mix of functional, automated, exploratory, regression, and performance testing,” he said. “Even testing organizations have to look for new tools and new ways to push quality.”

Test early and test often: It’s a phrase that almost every organization eats, sleeps and breathes these days, and there is a reason why organizations are so adamant in following it, according to Cochran. “It is a phenomenal way to ensure quality and ensure ownership from a team perspective,” he said.

In order to go about testing early and often, Cochran suggested doing exploratory testing. “Exploratory testing allows your QA people and even other parts of your organization to get inside of your product, even if [it is] unfinished and as development is occurring,” he said.

“And the fact that QA is involved early in the process makes them aware of the requirements and how things are changing throughout the process and not just at the end of the string.”

Go beyond test early and test often. “Testers have to be in constant communication with developers to know what changes are coming so that they can adjust test cases appropriately,” said Bracken. “You should try to automate when you can, but be smart about it.”

Bracken recommended doing performance smoke tests on CI builds multiple times a day, rather than conducting full load tests and leaving the larger more complicated tests for nightly builds or near the end of the sprint.

Test on real devices. “On mobile devices, there are interoperability concerns. For example what happens to an app’s functionality if you start testing?” said Jean Suplick, solution architect at Orasi Software. “So it is very important to test on real devices whenever possible.”

Testing on real devices provides quick, on-demand access to the different configurations of real devices, according to Suplick.

Consider the configurations. Testers should take into account the number of different device, operating system, carriers and network configurations an app needs to be tested on, according to Suplick. For mobile, the device and the network are especially worth considering, said Bracken.

“For performance testing, devices have different screen sizes and hardware capabilities, which dictate what content is delivered to that device and how fast,” he said. “Additionally, mobile network conditions including bandwidth, latency and packet loss can have a huge impact on application performance.”

Catching up to the speed of development
With software development getting faster and faster, everybody wants to be the first to release, and with end users becoming more and more demanding, updates happen almost every day. So SD Times asked experts: If development needs to be quick, fast and effective, how can testing catch up?

Jean Suplick, solutions architect at Orasi Software: The question implies that testing is something separate from development, which really isn’t the case. Even if by development you mean coding, then testing is already there in the form of unit testing. And certainly, if you’re doing test-driven development, the answer is plain.

In the agile world, though, the same types of testing we’ve done in the past still apply. You still need to do integration testing, as well as functional and non-functional testing (for instance performances/scalability and security testing), and end-to-end testing of the full business processes. You still need to do regression testing. The pressure is how to accomplish all the testing in the time allotted, how to efficiently test.

The solution is to explicitly allocate time to accomplish these as part of your release and sprint/iteration planning. Test automation will help quite a bit, so long as time is allocated for it in each sprint, and test automation goals are considered as vital as coding objectives. And if you’re trying to achieve Continuous Delivery, test automation is absolutely required.

Jonathan Bracken, vice president of marketing at Neotys: Communication with developers is essential. Testers need to know what changes to the application are coming down the pipe in order to be able to adjust test scripts and ensure a quality release at a fast pace. This, of course, is always easier when you’re using tools that automate some of the test design maintenance process.

Beyond communication and test design maintenance, automating non-functional testing steps (load and performance included) helps to ensure that every release not only functions, but performs well under load.

Kyle Cochran, vice president of product management at QASymphony: There are tools available that allow testers to exist in a more agile environment and still feel like they are releasing quality software, making their jobs more efficient and allowing them to evaluate more of the product without sacrificing that level of quality.

Agile in theory has everybody in one room or at a set of desks, and they are working through a number of stories and doing that in a face-to-face type of environment. But that’s the theory. I think the reality is that many software organizations are very distributed, and they have to find ways to communicate and operate efficiently with the pressures of developing software quality with high quality. And I think software organizations without the right tooling that allows good communication and good visualization of issues that exist or how well the product is performing are going to struggle. The proper tool makes it easier for both developers and testers to do their job.

Testing in an agile world
Trends like agile and tools that don’t require specialized skills have made it easier for developers to take on testing tasks, but that doesn’t mean testers lose their role or responsibility in this new world of development, according to Bracken. Developers have expanded their roles to take on testing, but their testing roles are limited to taking over test case designs and writing unit tests, he said.

“Even with this shift, the continued value testers can bring to modern organizations is in the analysis of test results,” said Bracken. “It’s easy to know what to do when the results are pass/fail, but with exploratory testing, stress testing, etc., deeper analysis can provide a lot more value when it comes to things like capacity planning and getting to the root cause of issues.”

Developers may be doing testing, but that doesn’t make them a tester. Testers bring with them a certain way of at looking at software that a developer may not be as suited for, according to Cochran.

Having developers take part in testing actually is an important part of making everyone on a team equal, according to Suplick.

“Testers—people who know and have experience with SQA principles—are just equal members of the team now, rather than being siloed and good testing being considered an afterthought,” she said. “They have an opportunity to bring their expertise to the cross-functional team as an equal partner, and I think that raises the overall awareness of QA in the team.”

A guide to testing tools
Applause: Applause provides in-the-wild testing services that go beyond the traditional QA lab to create a great experience for app users where they live, work and play. With functional, usability, localization, load and security testing, Applause services span the entire app life cycle. Through a global community of more than 150,000 professional testers, Applause empowers companies to understand how apps will work in the real world, with real people on real devices, across locations and use cases that match actual users.

Borland, a Micro Focus company: Borland has a variety of test tools that are open, agile and work across the entire application development life cycle. Its Silk Portfolio of testing solutions reduces cross-browser testing time, automates manual test processes, ensures peak performance, and streamlines testing across desktop, Web and mobile. It covers automated functional tests, performance testing in the cloud, application performance testing, collaborative test management, cross-browser testing, quality assurance, and mobile testing.

Coverity: Coverity provides advanced static analysis algorithms designed to detect critical defects specific to each programming language. Test Advisor—Development Edition finds and fixes more issues earlier in the life cycle, improves efficiency of automated testing, and reduces the risk of costly damaging software failures. Test Advisor—QA Edition helps QA teams improve manual and automated testing efforts for .NET and Java apps. Policy Manager helps users adopt development testing across their organization. Code Advisor detects quality and security defects in the development workflow.

dynaTrace: dynaTrace for Test empowers test and QA teams with an application performance testing solution for end-to-end transaction regression, component regression, load and scalability testing. It compares test runs, automatically identifies regressions and provides code-level details to triage issues. dynaTrace Web Load Testing delivers accurate cloud-based Web load testing, with one click “load to code” analytics. Web Load Testing generates high-volume load that exercises the full Web application delivery chain. It provides deep performance analytics to detect, isolate, troubleshoot and fix problems.

HP: HP provides a suite of software testing solutions that enable organizations to address the needs of modern application delivery from agile test tools that integrate into the developer ecosystem, to manual and automated functional/load testing that offers coverage for a broad range of technologies from ERP to Web and mobile. HP’s testing offerings also provide virtualization capabilities that allow customers to simulate service and network conditions, facilitating faster release cycles and application optimization.

IBM: IBM’s DevOps incorporates continuous testing into the full development life cycle, providing project teams with constant feedback, and enabling them to test earlier with greater coverage—removing traditional testing bottlenecks such as unavailable test environments. IBM’s comprehensive continuous testing solution covers UI and integration Test Automation, performance testing, and Service Virtualization, ensuring that customers can approach all of their DevOps testing challenges with confidence.

Keynote: Keynote offers a variety of testing solutions to help users ensure the quality of their mobile app and Web experiences before and after launch. For mobile testing, Keynote provides Mobile Testing Free to spot check mobile websites, Mobile Testing Pro to manual test mobile apps and sites and Mobile Testing Enterprise to automate mobile app and site testing. In addition, Keynote offers load testing solutions to test from the users perspective, delivery high-volume traffic on demand and accurately model interaction, arrival patterns and geographic diversity.

Microsoft: Microsoft provides comprehensive testing tools for developers, testers and quality assurance stakeholders. Visual Studio Test Professional 2013 with Team Foundation Server offers test planning, test case management, and manual and automated testing, and sends high-fidelity defect reports to cloud-based or on-premises ALM solutions. Visual Studio Online offers a load-testing service and Application Insights for monitoring application performance and exception events in any environment. Teams can also create and manage dev/test environments, automate builds with Continuous Integration, orchestrate release management, and automate deployments of their applications.

Neotys: Neotys helps companies improve the quality, reliability and speed of their Web and mobile applications using next-gen load and performance testing products and services. NeoLoad is a load and performance testing solution designed for Web and mobile applications to realistically simulate user activity and monitor infrastructure behavior. NeoLoad has fully integrated mobile device simulation and network emulation capabilities, and enables agile teams to automate testing with wizards, pickers, checkers and integrations with CI servers like Jenkins.

Orasi: Orasi is a leading provider of software testing services utilizing test management, test automation, enterprise testing, environment hosting and mobile testing technology. The company is laser-focused on helping customers deliver high-quality applications, no matter the type of application they’re working on and no matter the development methods or delivery processes they’ve adopted. In addition to its end-to-end software testing, Orasi provides professional services around testing, processes and practices as well as software quality assurance tools and solutions to support those practices.

Parasoft: For 27 years, Parasoft has researched and developed software solutions that help organizations define and deliver defect-free software efficiently. By integrating development testing, API testing and service virtualization, reduce the time, effort and cost of delivering secure, reliable and compliant software. The majority of Fortune 500 companies rely on Parasoft in order to produce top-quality software consistently and efficiently.

QASymphony: QASymphony offers a suite of testing tools built from the ground up for agile. Through its visual-based tools, QAS provides insight into exploratory, automated and scripted testing details. qTest eXplorer records and documents exploratory testing, and also saves time when performing traditional manual testing. QASymphony also provides qTest, a scalable test-management solution that allows organizations to coordinate, track and manage multiple software testing projects and teams across multiple locations in real time.

SOASTA: SOASTA’s cloud-based platform delivers key testing capabilities uniquely suited to the next generation of Web and mobile applications. It provides real-time analytics, visual test creation, cross-cloud grid provisioning, and the ability to start, stop, pause and restart tests. CloudTest allows testers to build, execute and analyze performance tests of any scale inside the dev lab, all the way to live production. TouchTest delivers mobile functional test automation capabilities for multi-touch, gesture-based mobile applications.

Telerik: Test Studio and Mobile Testing are sets of test automation tools that help teams be more efficient in regression testing, improving test coverage and reducing the number of bugs that slip into production. The company offers automation solutions for GUI, load and performance testing for desktop, Web, hybrid and native apps running on any device (Android/iOS/Windows Phone). Tests can be authored in C#, VB.NET or JavaScript.

TestingWhiz: TestingWhiz is a codeless test automation solution for Web and cloud applications. It performs functional test automation, mobile test execution, Web UI comparison, database testing, Captcha automation, automated regression testing, distributed test execution, keyword-driven testing and data-driven testing, and provides a scheduler to enable testers to run test scripts at a predefined time. There is an enterprise edition of TestingWhiz for large distributed test teams as well as a community edition for small test teams and individual testers.