Software testing is all about automation

Automation is just like every other technology in that it takes effort to be successful. It’s another strategy that can be incorporated into the software testing life cycle, but issues come from testers figuring out how they can build it into a process and determining what solutions or tools are right for their company.

Over the past few years, testing went from 100% manual to incorporating automation so tests could be fast and repeatable. With mobile, agile and DevOps as driving forces, the payoff and ROI for automation that was promised years ago is finally happening, according to Mobile Labs vice president of mobility solutions Dan McFall.

Since businesses are moving to faster releases, the time to market has changed. According to Anand Kamat, group program manager at Microsoft, getting a software idea to become reality in a short timeframe puts pressure on organizations.

Although the focus is on automation (and rightfully so), manual testing is still relevant and takes many different forms in agile, said Kamat. Developers are continuing to incorporate more automated tests, with “test early, test often” still the guiding principle.

The risks of automated testing
Automation isn’t a one-size-fits-all solution. Companies have to complement test automation metrics with data from manual testing, exploratory testing, user acceptance testing, and testing in real-world customer environments to form a comprehensive view about quality of the product, according to Anand Kamat, group program manager at Microsoft. He suggests considering the following risks before getting started:

• Most often automation does not cover 100% of use cases, but having a consistent pass rate of 100% can give you a sense of “false confidence” that is fatal.
• Code coverage is not a reliable metric for ensuring end-to-end quality, but is often used as a measure to gauge the effectiveness of test automation.
• For connected applications involving multiple components, your “one-box” setup is not a real-world scenario. If you are not testing in an “integration environment,” you are not testing with right dependencies.
• With frequent changes in user experience, ROI on UI Automation might be limited for multi-channel applications. Having a services/API testing strategy combined with exploratory testing might be a better alternative.

Speed over quality
Because of the rise of automated tests, the need for speed continues to stay on companies’ minds. Matt Brayley-Berger, worldwide product marketing manager for HPE, said that his company always likes to ask the question, “Would you like to have better quality or faster speed?” They pondered the question, eventually reaching a consensus that organizations are finding ways to remove a lot of barriers that would have already lowered quality, according to him. This means they can still focus on the speed of release without sacrificing quality. And, if testers became more technical and had to work with development a little more closely, it forces the DevOps conversation sooner.

“It’s not the solution, but maybe that behavior is creating an environment to have more productive conversations with evolving testing and evolving quality,” said Brayley-Berger.

The shortcuts that take place during the development process can take a one-month turnaround to a one-week timeframe by simply reducing the amount of testing that is being done. According to Walter Capitani, a product manager at Rogue Wave, this is what companies are doing. He said they take shortcuts in testing by reducing the tests or by doing things after they’ve finished the release because they figure if they find a quality problem, they can always “patch it later.”

“They are taking advantage of [having] an easy way to distribute your code, but the cost of distributing new code is not very high,” said Capitani.

He said he sees these shortcuts because the idea is that code can be fixed easily if something is found after the release. However, doing so “definitely compromises quality,” according to him. Compromising quality can lead to things like loss of revenue, people returning your products, or your brand being damaged, perhaps permanently. For these reasons, Capitani says it’s important to not sacrifice quality for speed.

Alon Girmonsky, CEO and founder of Blazemeter, said that speed and quality are two separate ideas. He said that people think that one comes in front of the other, but there are certain teams that are responsible for making sure the release is faster or the quality is better, so it’s “not like if you want quality you need to compromise on the speed.” He said that in the past, people said that speed and quality cannot live together, and if anyone thinks this to be true in this age of software testing, they are a “traditional player.”

“In the past, you could either do quality or speed; you had to compromise on the test coverage,” said Girmonsky. “You should not compromise your test coverage for speed.”

McFall said that people sometimes get caught up in one metric and not the other when they are talking about speed and quality, but the only way to protect yourself is not in concentrating on pre-release quality, but post-release monitoring of your quality.

“That’s why you see performance monitoring being important,” said McFall. “Embrace DevOps and monitor and manage. Have the ability to get that feedback into automation processes.”

Managing the risks is another way to make sure the company is achieving the speed and quality that it seeks. Agile teams have been embracing DevOps practices to monitor their production environments, detect failures in real time and mitigate them in an “accelerated fashion,” which has given them an idea of what to test before production, and what to monitor and mitigate in production, according to Kamat.

“Previously we used to emphasize on 100% test execution before release and believe that all bugs have been found,” he said. “Reality was much different, but the feedback loop took much longer and made correlation of such bugs to test execution difficult.”

Speed is just the way things are right now, and to some extent, that can’t be argued. But, as a QA team, you have to be vigilant and say that quality is still important, according to Jason Hammon, director of product management at TechExcel. He said it could cost a company tremendously if something bad happens, as there is no one to blame but the company for taking shortcuts.

Hammon said that having a solid process and the right tools are most important, and could help a company. Getting a test-management tool that allows a company to easily and quickly analyze results can also speed some processes up without losing the quality at release time.

Mobile devices and automation
The evolution of software testing isn’t the only reason automation is becoming a must for testers. Mobile devices have changed the software game because it’s moved the control of the platform away from the people who are making the application, and putting it into the hands of mobile vendors and users, according to Michael Ryan, CTO of Mobile Labs.

He said that if you wind back the clock to pre-mobility, some of the most advanced shops in the world were doing automated testing. Now, mobile is making automation a must because of the devices that need testing. Generally speaking, he said that the number of devices you test on correlates linearly with application quality.

“The fact that it works on one Android phone is interesting, but you need to look at different versions of the operating system and different form factors just to cover the basics,” said Ryan. “In a way, the more devices you test on, the higher your level of quality is going to be.”

Automation really became popular when mobile was introduced because of the variety of platforms, according to McFall. With desktop, a simple case like running a login script would only take a few tests. With mobile, this simple test case means increasing the test coverage because of the amount of mobile platforms. The only way to keep up with mobile is with automated testing, and for most enterprises, it’s the cheapest, most logical way, he said.

Looking back 20 years, companies that talked about automating testing said that in order to justify the effort, you needed a large number of test cases, according to Ryan. Now it’s not about whether there are enough test cases, but instead you reduce the number of test cases that will bring the company the greatest return, he said.

“Point being, the number of test cases is now automatically large enough that automated testing is not just justified; it’s a survival technique I think,” said Ryan.

Tools and solutions
One major suggestion for getting started with automation in an organization is to first have some sort of plan and determine what tools or solutions are best for the company and its goals. For some companies, an open-source automation tool could be best, or an off-the-shelf tool could be better. It could be a combination of both, but it’s important that teams sit down and consider the tradeoffs.

“There is no such thing as a free lunch,” said Mobile Labs’ McFall. “If it’s free, there are [going to be] different levels of customer service, documentation, management, visibility, security compliance checks, and these are the things you need to contemplate.”

McFall said he is excited for the open-source movement, and he said that it actually accelerated people bringing in technology for automation. One of the major disappointments he has seen from customers or people in the industry is that an open-source automation tool is tricky to set up. The level of support for the tool is different, the knowledge base is different, and there might not be management tools to go along with the automation.

“I have seen organizations struggling with that,” said McFall. “People should look at all the options they have for automation.”

He added that it also helps to look at the skillsets of your employees before choosing a tool. He gave the example of an open-source automation tool needing C++ when the company’s employees know Java. The employees would then need to learn C++, and the tool could still strategically fit into the process, but these are the unintended costs of change that companies do not think of.

TechExcel’s Hammon said that there is definitely a need for open-source tools, as they can be effective for companies as long as they manage them without allocating resources just to maintain them.

“One thing you do get from licensing software is that if you know you are working with a responsive vendor, it will reduce the resources you need to manage the tools,” said Hammon.

Besides open-source tools, the addition of new tools in general has also contributed to the evolution of software testing. In the past, testing could involve executing a test case where the humans do all the writing of the cases, and coding those test cases means spending almost as much effort writing test cases as you do code, according to Rogue Wave’s Capitani. Now, there are tools that can help, such as those that can automatically generate test libraries and allow testing of the code to make sure it doesn’t do anything it shouldn’t be doing.

Capitani said that if companies are not doing any automation and are testing software, then they “are kind of behind the eight ball.” Automation is not a new idea, but it was a few years ago.

“No one is testing software completely by hand any more, [although] there are some,” said Capitani. “The reality is any modern software that is being developed should have some form of automated testing.”

What automation tool is right for your company?
Hung Nguyen, CEO of LogiGear, said a company that is looking at automated testing needs to first figure out how much automation they need and in what timeframe. He said it’s a matter of “taste” and capability, and every tool has its pros and cons.

• What is the current capability of the company and the test team?
• Where will the automated tests be done?
• Who will do the automated tests? Will it be developers or testers?
• Where is the allocation of resources?
• Where will the rest of the team come from? Will it be in-house or an outside party?

The traits of a modern tester
The push toward faster releases is breaking down the walls between testers and QA pros, and all teams are starting to have more lines of direct communication, allowing testers to have a greater breadth of knowledge. In the past, the testing team would just kick something over the wall and wait for some sort of return, but now all teams are getting together and having long conversations about production and defects. This is just one skill that modern testers will need to keep up with automation.

HPE’s Brayley-Berger said that along with being able to have conversations about what they are doing, modern testers are being forced to become more technical and are actually taking more of a pure subject-matter expertise role rather than primarily testing. The tester role is far more technical than before as well.

The best testers he sees are the ones that can be technical in the testing realm, but still understand things from a business perspective. He said the customers he sees as being the most successful were the ones where the testers had the ability to do things like scripting and coding, but were also able to understand business rationale.

“The tester knows that there are certain business functionalities that are very critical to a certain release or a certain competitiveness,” said Brayley-Berger. “They’ll be able to speak up earlier in conversations when we maybe have to add certain quality elements or architectural elements.”

Hammon said that he thinks companies want individuals that really understand automation. Being able to respond to the challenges of getting things done and also trying to find efficient ways to do things is a strategy that they will be able to develop.

Testers should also understand customer end-to-end scenarios as well as possess analytical abilities, critical insight, and an eye for detail, according to Microsoft’s Kamat. He said they should embrace change—like automation—instead of feeling threatened.

Besides having both business and technical skills, Rogue Wave’s Capitani said that testers should be good at breaking things—as in, they should become good at recognizing vulnerabilities or weaknesses in code design because they have been incorporated into the software development life cycle.

“They’re good at reverse-engineering how code works, so a good tester needs to be able to find the root cause of problems,” he said.

Testers should also become familiar with the different automation tools that exist. Hung Nguyen, CEO of LogiGear, said that testers should have knowledge of tools like Selenium, Cucumber, TestingBot, and other third-party tools. All of these methods mean different frameworks, and you have to prescribe what the automated testing tool is supposed to be doing.

“Software changes rapidly; it means every time the software changes your tests will be broken and you need to fix them,” said Nguyen. “[Exploring] these different methods will help them.”

Knowledge of automation is one of the best traits that a tester could have, according to Blazemeter’s Girmonsky, because today it’s at the “heart of everything,” unlike manual testing. He added that automation is the one thing he recommends anyone adding to their portfolio because automation is not only for testing, but also for the Continuous Delivery and Continuous Integration.

“People may call this the continuum; this is a different name for automation,” said Girmonsky. “Whenever you [test] anything in the context of continuous, it means it has to be automated.”

Room for improvement
Agile environments require a quick turnaround for releases, and the movements of Continuous Delivery and DevOps, along with application life-cycle management, all contribute to the rapid release of software.

Manual testing is no longer a necessity, but it will never be removed from software development, according to Nguyen. Other experts said that software development could improve significantly by automating tests, or by having teams facilitate agile practices.

“Today, for any organization or business to survive in a competitive environment, it needs to adopt agile methodologies. It has to,” said Girmonsky. “This is agile, automation, Continuous Delivery, and the new age software delivery methods.”

Brayley-Berger said that a major struggle organizations are still dealing with is solutions for software development. In order to effectively test and mitigate some of the mishaps that happen when organizations get closer to releases, there needs to be a bit of embracing the concept of virtualization, which again ties into that mantra of test early and often.

“There is certainly a benefit around doing levels of virtualization to at least provide easier access to people so they can do more testing earlier,” said Brayley-Berger.

Teams should also consider all aspects of software testing, including unit, functional, integration, load, performance, manual, and others, according to Microsoft’s Kamat. He said that testers should never believe that being overweight on one aspect of testing would make it acceptable to completely ignore other testing strategies.

“Historically, testers have been rewarded for being conservative, and so have not seen any examples of testers taking major risks,” said Kamat. “However, current DevOps practices are enabling us to take more risks by leveraging the tools at our disposal.”

What are your tips for testers getting started with automation?
Since automation isn’t new to the software testing life cycle, companies might be set in their ways when it comes to automated tools, solutions or strategies. SD Times spoke with some experts about what tips they have for companies who are looking to start automation.

Matthew Brayley-Berger, worldwide product marketing manager at HPE: Automated testing is one of those things that really help a team save time (and realize much higher quality), when they’re done correctly. Of course, the flip-side is that they can also become a huge time-waster when not. When it comes to Functional/Regression testing, I’ve always found that the teams that start by getting an appreciation for what the core architectural components of their application are likely to be first, and then planning automation to support the most manually taxing (and error-prone) tests. This may also involve an early plan of the expected tests, and manual execution steps (initially), which can later be used to develop automation. The earlier the better though, when it comes to any automation (even at the unit test level), because often there are constraints that need to be built into any solution in order to make automation smoother.

Dan McFall, vice president of mobility solutions at Mobile Labs: What I personally see as a common “mistake” for those just getting started with automation is trying to automate “everything.” Typically new automation engineers want to get the entire test suite automated, and there are simply lots of tests that don’t make sense to automate (for example, user experience tests). I tend to recommend focusing on the most repeatable tests (i.e. the “boring” tests) and then the highest-risk areas of your applications (e.g. balance transfers, room check-in, ticket purchase, etc.). This allows for manual testing to be more exploratory and contextual than just focusing on “does my login work?”

Alon Girmonsky, founder and CEO of Blazemeter: I recommend treating automation as a strategy as opposed to a task. The question shouldn’t be how do I automate a handful of existing tests. The question should be how do I create an end-to-end automated process that yields between 85% to 100% test coverage. I’d use open-source tools and frameworks to build the process, and use my best engineer to architect and build it.

Anand Kamat, group program manager and product owner of Visual Studio Team Services at Microsoft: If we broaden the automation context, there are many places where testers can get started with automation and have an immediate impact on efficiency, such as automation of test environments, leveraging cloud environments for faster execution, automation of end-to-end scenarios, monitoring application performance and usage telemetry, and leveraging scripting skills to write extensible tasks for many popular CI systems. As they build new skills in automation, they should continue to leverage their strengths in manual and exploratory testing to continuously improve quality of the product.

Walter Capitani, product manager at Rogue Wave: To get started, do one of two things: hire yourself an automation test team and select a tool that’s appropriate and start deploying it. Second, design to a certain degree your code for automation. You want to start looking at your source code and adjust the architecture of the source code so it’s suitable to automation.

Jason Hammon, director of product management at TechExcel: Outlining your automation goals is a great way to get started. Determine your goals before you try to automate. Automation can be a significant benefit for any QA team, but without having a plan and targeting specific results, it can be overwhelming. Starting small and focusing on an area or goal that is well suited for automation can lead to a quick win.

Hung Nguyen, CEO of LogiGear:

Know your…

• Goals: Automation is a journey and long-term commitment. Be realistic.
• Capability: Test design, programming, and domain expertise are essential.
• Tests’ design/authoring method: BDD, keyword-driven, ABT, etc. Focus on rapid test creation and low maintenance.
• Framework: Is it robust, easy to use for authoring, low maintenance, and collaborative?
• Tool options: Open source or proprietary, free license of paid, deep or light programming requirements, steep or light learning curves, and expensive or cheap support costs.

A guide to software testing tools
Applause: Applause empowers companies to deliver great digital experiences to their users by providing in-the-wild testing services that go beyond the traditional QA lab. Applause offers functional, usability, localization, load and security testing. The company’s testing services span the entire app life cycle and include access to its global community of more than 200,000 professional testers in more than 200 countries and territories. Applause also provides a suite of complete app quality offerings, including test automation, mobile beta management, and mobile sentiment analysis that help companies achieve the 360-degree app quality they need to thrive in the modern digital economy.

BlazeMeter: BlazeMeter ensures delivery of high-performance software by enabling DevOps teams to quickly and easily run open-source based performance tests against any mobile app, website or API at massive scale to validate performance at every stage of software delivery. The rapidly growing BlazeMeter community has more than 100,000 developers and includes prominent global brands such as Adobe, Atlassian, Gap, NBC Universal, Pfizer and Walmart as customers. Founded in 2011, the company is headquartered in Mountain View, Calif., with its research and development in Tel Aviv.

HPE: HPE Software’s Functional Testing solutions help to deliver high-quality software while reducing the cost and complexity of functional testing. HPE’s solutions address the challenges of testing in agile and Continuous Integration scenarios, as well as hybrid applications, cloud and mobile platforms.

LogiGear: LogiGear provides a variety of testing solutions, including test automation and technical testing for software applications. With the no-coding and keyword-driven approach to test authoring in its TestArchitect products, users can rapidly create, maintain, reuse and share a large scale of automated tests for desktop, mobile and Web applications. Founded in 1994, LogiGear has completed software testing and development projects for prominent companies across a broad range of industries and technologies.

Microsoft: Microsoft provides a specialized tool set for testers, which delivers an integrated end-to-end experience starting from agile planning, to test and release management on premises or in the cloud. Microsoft test tools eliminate waste across the application life cycle and embrace change through the use of test impact analysis, rich actionable bug filing, manual testing, exploratory testing, user acceptance testing, automated testing and load testing. Use your favorite browser to easily access the central test hub to gain greater visibility and end-to-end traceability, which allows for better decision-making. Align your test teams with the application life cycle by facilitating effective collaboration between development, test and operations, to turn DevOps into reality. Microsoft empowers testers by bringing technologies like Lab Management and Release Management to extend its capabilities to virtualized lab environment management to deliver high-quality applications faster.

Mobile Labs: Mobile Labs’ enterprise-grade device cloud, deviceConnect, provides secure remote access to mobile devices for managed use by developers and testers, enabling automated continuous quality integration, and automated or manual app/Web/device testing. With deviceBridge (an extension to deviceConnect), many test automation frameworks and developer tools used for checkout and debugging can retain cloud-based devices as if locally connected by USB. deviceBridge serves as a “virtual USB cable,” connecting cloud-based devices to a local laptop or server over Ethernet. A certified add-in to HP Unified Functional Testing (UFT) software, Mobile Labs Trust enables testers to cost-effectively extend existing HP functional testing solutions to test mobile apps on real devices without jailbreaking or rooting.

Neotys: Neotys is the leading innovator in Continuous Performance Validation for Web and Mobile applications. Neotys load testing (NeoLoad) and performance monitoring (NeoSense) products enable teams to produce faster applications, deliver new features and enhancements in less time and simplify interactions across Dev, QA, Ops and business stakeholders. Neotys has helped more than 1,600 customers test, monitor and improve performance at every stage of the application development life cycle, from development to production, leveraging its automated and collaborative tooling.

Orasi: Orasi is a leading provider of software testing services, utilizing test management, test automation, enterprise testing, Continuous Delivery, monitoring, and mobile testing technology. The company is laser-focused on helping customers deliver high-quality applications, no matter the type of application they’re working on and no matter the development methods or delivery processes they’ve adopted. In addition to its end-to-end software testing, Orasi provides professional services around testing, processes and practices, as well as software quality-assurance tools and solutions to support those practices.

QASymphony: QASymphony offers a suite of software-testing tools purpose built to help agile development teams improve speed, efficiency and collaboration. qTest is a Test Case Management solution that provides a better way for companies to centralize and manage test cases. qTest is robust, easy to use, and integrates with the most popular tools used by agile development teams like Atlassian’s JIRA, VersionOne and CA Agile Central. Additionally, QASymphony offers qTest eXplorer for teams doing exploratory testing. qTest eXplorer lets a tester record everything he is doing during the testing session and automatically creates detailed documentation of any software issues. This eliminates the need for the tester to do tedious manual documentation, thus saving significant time. QASymphony’s newest innovation is qTest Scenario, the only JIRA add-on that helps teams optimize and scale Test First methodologies (BDD, TDD, ATDD) across their organization. QASymphony has more than 300 customers, including Salesforce, Barclays, Adobe, Samsung, Office Depot and Dell, and was named a “Cool Vendor in Application Development” by Gartner.

Rogue Wave: Rogue Wave is the largest independent provider of cross-platform software development tools and embedded components in the world. Through decades of solving the most complex problems across financial services, telecommunications, healthcare, government, academia, and other industries, Rogue Wave tools, libraries, and services enable developers to write better code, faster. Product lines include Klocwork static code analysis, OpenLogic open-source audit and support, CodeDynamics for dynamic analysis, and Zend Server for professional PHP distribution.

Sauce Labs: Sauce Labs provides the world’s largest cloud-based platform for automated testing of Web and mobile applications. Its award-winning service eliminates the time and expense of maintaining an in-house testing infrastructure, freeing development teams of any size to innovate and release better software, faster. Optimized for use in CI and CD environments, and built with an emphasis on security, reliability and scalability, users can run tests written in any language or framework using Selenium or Appium, both widely adopted open-source standards for automating browser and mobile application functionality. Videos, screenshots and HTML logs help pinpoint issues faster, while Sauce Connect allows users to securely test apps behind their firewall.

SOASTA: SOASTA’s Digital Performance Management (DPM) Platform enables measurement, testing and improvement of digital performance that predicts and diagnoses performance issues before they occur. It includes five technologies: mPulse real user monitoring (RUM); the CloudTest platform for continuous load testing; TouchTest mobile functional test automation; Digital Operation Center (DOC) for a unified view of contextual intelligence accessible from any device; and Data Science Workbench, simplifying analysis of current and historical Web and mobile user performance data.

Synopsys: Through its Software Integrity platform, Synopsys provides a comprehensive suite of best-in-class software testing solutions for rapidly finding and fixing critical security vulnerabilities, quality defects, and compliance issues throughout the SDLC. Leveraging automation and integrations with popular development environments and tools, Synopsys’ Software Integrity Platform empowers customers to innovate while driving down risk, costs, and time to market. Solutions include static analysis, software composition analysis, protocol fuzz testing, and interactive application security testing for Web apps.

TechExcel: DevTest, from TechExcel, is a sophisticated quality-management solution used by development and QA teams of all sizes to manage every aspect of their testing processes. Within a single application utilizing a wizard-driven interface and linked to requirements, users can create and manage release and test cycles, plan and assign test tasks, execute test coverage, and submit product defects. Built-in, dynamic reporting, tracking and analysis also provide complete accountability and quality assurance across all stages of the development life cycle. DevTest addresses the needs of contemporary testing challenges for faster delivery of quality software.

Telerik: Test Studio is a test-automation solution that helps teams be more efficient in functional, performance and load testing, improving test coverage and reducing the number of bugs that slip into production. The company offers automation solutions for desktop, Web and mobile testing, which includes hybrid and native apps running on any device (Android or iOS). Tests can be authored in C#, VB.NET or JavaScript.