Companies today are like teenagers learning how to drive; there’s a lot of stop and go, and a strong desire to go fast. While it’s necessary for organizations to find a solution that fits their agile transformation, experts suggest there is no need to rush. Before getting the green light to go to market, teams should slow down and take a step back so they can avoid some major automation pitfalls.
Like anything with software testing, there is risk. Automating tests takes effort in order to be successful, and the last thing business leaders want are poorly conceived and executed tests, creating a costly and time-wasting mess.
Lubos Parobek, vice president of product at Sauce Labs, said a major risk he sees companies concerned with is the idea of test flakiness, where tests fail because they were badly written. This is why best practices are important, he said, because the more teams are educated, the better chance they have at building tests that are maintainable and reliable.
(Related: Software testing is all about automation)
Michael Eckhoff, software test automation veteran at Tricentis, said one of the biggest pitfalls he sees is companies trying to pick a software automation tool and expecting it to solve all of their problems. What companies really need to do is figure out how to pick the right tool that best meets their needs, and to ensure that the company is ready for it, according to him.
“You can typically get that first quick win with intuitive test-case design,” said Eckhoff. “Getting to that second phase of really intelligently designing your test portfolio to ensure that you are testing to get the best risk coverage or best business value is where that [best-practice] methodology comes back into play.”
Another risk comes from overconfidence, said Eckhoff. Companies look at their processes, see a large amount of automated test cases, and assume they are in “good shape,” he said. The problem in this case is the tests are not clearly defined, and in many cases, the test definition doesn’t stay consistent with what the test is actually doing, he said.
Companies need to get their teams to buy into the concept of test automation, or organizations will continue to treat agile as the “Wild West,” where teams do whatever they want just to release things fast, and then when it breaks, they do it again, said Eckhoff. He said this way of thinking just doesn’t work for the enterprise, and those organizations need more than just a tool to achieve speed and quality applications; they need a process in place that supports agile development and QA early on.
Rushing to automation
When getting started with a software automation tool or solution, organizations should pause and do a bit more planning up front before jumping into the deep end, according to Parasoft’s CMO, Marc Brown. He said he has seen organizations select an automation tool based on a very limited developer group, but in the end, the assessment wasn’t broad enough to see how it impacted their daily activities or how it helped their agile initiatives.
Brown also said that a lot of teams get into problems with their testing strategies because they rush into using an automation tool to quickly solve one problem, which means they lack the “big picture,” or what software automation can do over time for their company.
“I think if operations slows down and they get guidance from people who have expertise in agile development [and] Continuous Testing, and have looked [at the tool] broadly to see how it supports the developers, the testers, the managers, and the executives,” said Brown, “most people will recognize there is value in getting a solution that covers all of those various roles.”
Brown added that the one shortcut he sees companies taking is not testing adequately. He said these companies “race to market with blinders and really cross their fingers hoping they won’t get caught up with a bad defect.” The risk of not testing can be as detrimental as hurting the company’s brand or market reputation. And in some cases, having a malfunction in software could really physically hurt the customer.
Parobek said that he also sees companies trying to cut corners, and in his personal experience, it’s never worth it.
“Consumers more and more have very high expectations of web and mobile apps that are coming from the services they use, whether that’s banks, retailers, etc., so having the application that you cut corners with that you deliver early and it is buggy, it’s a bad tradeoff,” said Parobek. “Going to Continuous Delivery, Continuous Integration, and [automated] testing is a way you can speed everything up and get higher quality.”
He does believe that with the right tool, companies can “have your cake and eat it too,” but that requires proper investment and planning to do it right.
What testing pitfall does your solution aim to solve?
Marc Brown, software industry expert, Parasoft
Parasoft helps organizations perfect today’s highly connected applications by automating time-consuming testing tasks while providing management the analytics necessary to focus on what matters.
We also help developers become highly efficient at software validation using various technologies and techniques—from static and runtime analysis, to unit testing, to API testing all on a scalable software virtualization platform. Furthermore, we help the management team focus on what matters through highly unique data analytics and reports. By doing so, the team eliminates issues that will impact quality, security, performance, reliability, and customer satisfaction.
Parasoft solutions help teams eliminate defects early in development minimizing cost and rework, and focus on the risky areas of an application, saving time while improving quality and lowering security threats.
Lubos Parobek, vice president of product, Sauce Labs
The right tools are critical for a successful outcome when embarking to adopt continuous delivery and automated testing. For example, many organizations struggle with getting the required test coverage across platforms, while keeping testing from becoming a bottleneck in their delivery pipeline.
Many also struggle with higher error rates and flaky tests, particularly when the flakiness is caused by a failure in their in-house test infrastructure, not the test code itself. Sauce Labs can dramatically increase the speed, coverage and accuracy of automated testing. For example, Sauce Labs allows our customers to run tests at much higher levels of concurrency, with better coverage and with lower error rates than most can achieve on their own. We run over 1.3 million tests a day across 800+ OS, Browser and Device combinations.
Gerd Weishaar, vice president of product management, Tricentis
It’s become increasingly difficult for teams to build a test suite that not only covers the risks that are most critical to the business, but also is maintainable and efficient enough to be used for continuous testing of rapidly-evolving applications. Delivering positive user experiences requires continuous testing—and accelerated development cycles mean there’s less time available for designing, creating, updating, and executing tests.
Our solution is designed to help testers rapidly identify and create the tests that are most important for protecting key end-to-end business processes. Based on many years of R & D focused on Model-based Test Automation, we’re able to scan the application under test to help testers understand and define the optimal set of test cases that will cover their organization’s highest priority risks. The resulting test suite is perfectly-suited for continuous execution as part of the delivery pipeline, and it’s easily updated as the application evolves.
All this is accomplished without coding or scripting. Scripted tools lead to long complex test scripts, which get “fixed” in the script if technical challenges arise. At some point these scripts become too many, too complex and end up as maintenance nightmare. This is typically the reason why manual testing is still so popular (80% of test cases) after 20 years of (script based) automated test tools like QTP/UFT.
Rod Cope, CTO, Rogue Wave
Developers and QA engineers have to consider what might go wrong with code, but they aren’t able to anticipate everything. Despite best efforts, they’re building in and shipping bugs and security vulnerabilities with code.
Klocwork, our automated static analysis tool, enables software teams to find and fix bugs early in the life cycle, which saves time and money while speeding software delivery. It continuously monitors every line of code so errors can be identified that developers and QA engineers miss. Klocwork can be integrated seamlessly with continuous integration environments so issues can be identified and remediated prior to a build. As you’re writing code, it will tell you if you’ve introduced an error so you can fix it immediately. Klocwork also improves workflows and enforces best practices by enabling software teams address issues earlier in the life cycle.
Klocwork users are able to prevent security exploits and compliance issues that can negatively impact brands, customer relationships and profitability.
Survey: Old automation tools won’t work with DevOps
“The QA Wall,” or a common bottleneck for organizations that are adopting DevOps processes, is the current bottleneck of of 54% of DevOps adoptees, according to a recent survey from Appvance.
Moving fast and improving DevOps tools is possible, but companies need to start by reevaluating their QA automation processes and technology, according to Kevin Surace, CEO of Appvance.
“Every company is already going as fast as they can with their old tools, most of which are 10 to 20 years old in terms of architecture and were not designed for modern applications in a DevOps rapid environment,” said Surace. “If you hit a wall, new technology is one way to break down that barrier to move 10x or even 100x faster without abandoning quality.”
Seventy-nine percent of IT decision makers also reported that increasing the frequency of their releases is a top priority, and the same percentage agreed that increasing the frequency of their release could help them achieve their business objectives. The problem is, according to Surace, that companies will continue to struggle with the QA Wall by using old automation tools that can’t move as fast as DevOps.
“Until the silos created by those tools are eliminated, the QA Wall will continue to impede software release schedules,” he said.
A guide to automated testing tools
Applause: Applause delivers unmatched in-the-wild testing, user feedback and research solutions by utilizing its DX platform to manage communities around the world. The company’s testing solutions span the entire app lifecycle and include access to its global community of more than 250,000 professional testers in more than 200 countries and territories as well as access to specific, client-requested digital users.
CA BlazeMeter: The solution ensures faster delivery of applications by enabling DevOps teams to quickly and easily run performance tests on-premises, in the cloud or on virtual private clouds against any app, website or API at massive scale to validate performance at every stage of software delivery.
HPE Software: HPE Software’s automated testing solutions simplify software testing within fast-moving agile teams and for Continuous Integration scenarios. Integrated with DevOps tools and ALM solutions, HPE automated testing solutions keep quality at the center of today’s modern applications and hybrid infrastructures.
LogiGear: With the no-coding and keyword-driven approach to test authoring in its TestArchitect products, users can rapidly create, maintain, reuse and share a large scale of automated tests for desktop, mobile and web applications.
n Microsoft provides a specialized tool set for testers that delivers an integrated experience starting from agile planning to test and release management, on premises or in the cloud.
Mobile Labs: Mobile Labs’ deviceConnect provides secure remote access to mobile devices for managed use by developers and testers. With deviceBridge (an extension to deviceConnect), many test automation frameworks and developer tools used for checkout and debugging can retain cloud-based devices as if locally connected by USB.
Neotys: Neotys load testing (NeoLoad) and performance monitoring (NeoSense) products enable teams to produce faster applications, deliver new features and enhancements in less time and simplify interactions across Dev, QA, Ops and business stakeholders.
Orasi: Orasi is a leading provider of software testing services, utilizing test management, test automation, enterprise testing, Continuous Delivery, monitoring, and mobile testing technology.
Parasoft: Parasoft researches and develops software solutions that help organizations deliver defect-free software efficiently. To combat the risk of software failure while accelerating the SDLC, Parasoft offers a broad test. Parasoft’s enterprise, mobile, IoT, and embedded development solutions are the industry’s most comprehensive—including static analysis, unit testing, requirements traceability, coverage analysis, API testing, dev/test environment management, service virtualization and more. The majority of Fortune 500 companies rely on Parasoft in order to produce top-quality software consistently and efficiently as they pursue agile, lean, DevOps, compliance, and safety-critical development initiatives.
Progress: Telerik Test Studio is a test-automation solution that helps teams be more efficient in functional, performance and load testing, improving test coverage and reducing the number of bugs that slip into production.
QASymphony: QASymphony’s qTest is a Test Case Management solution that integrates with popular development tools. QASymphony offers qTest eXplorer for teams doing exploratory testing. qTest Scenario is a JIRA add-on that helps teams optimize and scale Test First methodologies across their organization.
Rogue Wave: Rogue Wave is the largest independent provider of cross-platform software development tools and embedded components in the world. Rogue Wave Software’s Klocwork boosts software security and creates more reliable software. With Klocwork, analyze static code on-the-fly, simplify peer code reviews, and extend the life of complex software. Thousands of customers, including the biggest brands in the automotive, mobile device, consumer electronics, medical technologies, telecom, military and aerospace sectors, make Klocwork part of their software development process.
SOASTA: SOASTA’s Digital Performance Management (DPM) Platform enables measurement, testing and improvement of digital performance. It includes five technologies: mPulse real user monitoring (RUM); the CloudTest platform for continuous load testing; TouchTest mobile functional test automation; Digital Operation Center (DOC) for a unified view of contextual intelligence accessible from any device; and Data Science Workbench, simplifying analysis of current and historical web and mobile user performance data.
Synopsys: Through its Software Integrity platform, Synopsys provides a comprehensive suite of testing solutions for rapidly finding and fixing critical security vulnerabilities, quality defects, and compliance issues throughout the SDLC.
TechExcel: DevTest is a sophisticated quality-management solution used by development and QA teams of all sizes to manage every aspect of their testing processes.
Sauce Labs: Sauce Labs provides the world’s largest cloud-based platform for automated testing of web and mobile applications. Its award-winning service eliminates the time and expense of maintaining an in-house testing infrastructure, freeing development teams of any size to innovate and release better software, faster. Optimized for use in CI and CD environments, and built with an emphasis on security, reliability and scalability, users can run tests written in any language or framework using Selenium or Appium, both widely adopted open-source standards for automating browser and mobile application functionality. Videos, screenshots, and HTML logs help pinpoint issues faster, while Sauce Connect allows users to securely test apps behind their firewall.
Tricentis: Tricentis is recognized by both Forrester and Gartner as a leader in software test automation, functional testing, and continuous testing. By enabling test automation rates of over 90%, we help Global 2000 companies control business risk as they adopt DevOps, Agile, and Continuous Delivery. Our integrated software testing solution, Tricentis Tosca, provides a unique Model-based Test Automation and Test Case Design approach to functional test automation—encompassing risk-based testing, test data management and provisioning, service virtualization, API testing and more. Tricentis’ 400+ customers include global names from the Top 500 brands such as A&E, Allianz, BMW, Deutsche Bank, HBO, Lexmark, Orange, Starbucks, Toyota, UBS, Vantiv, Vodafone, and Zurich Insurance.
