DevOps has transformed the way businesses think and software development teams work, but the power of DevOps is still limited.

According to Shamim Ahmed, CTO for DevOps solutions at the global technology company Broadcom, testing still stands in the way of achieving true DevOps and continuous delivery. Testing is a time-consuming process that requires many moving parts to happen in the right way, he explained.

Testing in general is just more complex, according to Maya Ber Lerner, CTO of Quali, a cloud automation and digital transformation company. “Testing is not as lightweight as development. You need to have you test automation in place, you need to have your applications in place, third-party components in place, and you need to have the right infrastructure and data set in place. Each one of those things can easily fail a test,” she said. 

RELATED CONTENT:
How do you help test in DevOps?
A guide to DevOps testing tools

In addition, there are just a number of different questions and scenarios you have to ask yourself when it comes to testing, Matt Davis, managing director for QA Systems, a software quality company said. “Testing, of course, is at the heart of something that you develop. You test it to determine whether it’s going to be released, but that testing can be on the basis of what should I test? Should I test everything? Should I be looking at impact analysis and change-based testing? Should I be looking at auto-test case generation?” he said. And these are just a small set of questions and thinking that is required for testing.

Luckily, there are some ways teams can start to break down the barriers of testing in DevOps:

Don’t treat testing like a phase in the life cycle.

Despite efforts to test early and to test often, Broadcom’s Ahmed said testing is still looked at as a particular phase in the life cycle, “when in fact it should be continuous and embedded throughout the entire life cycle,” he said. 

To do this, testing needs to shift left and shift right. “The more testing, especially around automation, that you can build into the application from day one is key,” said Dan McFall, CEO of Mobile Labs, an enterprise mobile app testing company. McFall explained techniques like test- driven development or behavior-driven development help developers become more involved in testing and really taking the time to look at the features and solutions, how those are going to be tested, and how those tests are going to validate business requirements. 

Shifting right enables developers to work better with operations to understand what is going on in production and take advantage of that to improve tests and test conditions, according to Ahmed. 

Acquire the right skills.

“As we shift left and shift right with continuous testing, we need to start to bring in additional skills to the traditional QA testing mindset,” said Ahmed. “For example, with shift left you need to bring in more developer skills, and developers need to be more adept to the concepts of continuous test automation and how to integrate automation into the development process through continuous integration. The mindset also has to shift because quality is now everyone’s responsibility. 

One trend Ahmed is seeing is the introduction of the software development engineer in test, which are individuals who participate in development and also know testing techniques like white box testing. “These guys are able to participate with developers for example in code reviews, understanding what is going to be built, and participating in the technical debt of the code,” he said. 

In addition to skills, it is equally as important that DevOps teams actually understand what is happening such as the assessment of code, how tests are impacted, how tests are built and rerun, how to automatically generate tests, according to Davis. “People need to understand the main criteria. Can it be automated? Is it an open platform? Can you integrate it with different things? The critical challenge is does it deliver, is it accessible and is there support for it,” he said. 

Make testing automated and continuous. 

“DevOps as a process doesn’t work unless it’s automated and continuous,” said QA Systems’ Davis. He explained that in order to shift from doing nightly builds to continuously integrating testing throughout the entire life cycle, things need to be 100% automated. “You can’t do things continuously if you have to manually intervene,” he noted. “You need to be able to trigger things and react to activities or react to results and outcomes all the way along your pipeline. 

One of the biggest door openers for automation is the connection on a pipeline, according to Davis. “There is a major advantage to being able to set up a series of quality gates and triggers, and automating different types of activities throughout the testing and development life cycle,” he said. “The more that can be automated in these pipeline stages via decorative pipeline scripts, the better it can be.” 

Test environments should also be automated, according to Quali’s Ber Lerner. “If you have a testing organization that’s trying to write tests real fast, and they are automated tests, but it takes two weeks or even three days to set up a test environment, are you really Agile?” she asked. Additionally, test environments should be set up in a way that separates environment issues from test issues so there is no confusion. 

Broadcom’s Ahmed explained another way to speed up testing is through model-based testing, a technique where tests are automatically generated by models or a description of the system’s behavior. According to Ahmed, model-based testing helps auto generate test assets on the fly to improve test productivity and make sure all the assets are available as soon as the requirements are ready.  

Impact analysis can also be used to speed up testing. Impact analysis, or change-based testing, helps run only the tests that were impacted by code changes, limiting the amount of tests that are actually run, according to Davis. 

“Quality counts for the bottom line. The way you make that bottom line more efficient is through automation and integration,” said Davis. “If software doesn’t work as well as it should, or the quality level is just not there, companies are going to lose ground to their competitors.”

Leverage manual testing.

Despite the increased need to automate things, there is still a need for manual testing in DevOps. Automated test assets free up testers to be able to do more value added pieces of testing such as testing from a real-user customer experience perspective or doing exploratory testing on a new piece of functionality, according to Broadcom’s Ahmed.

He explained that not everything can be automated and testing things that have a human element to it is hard to automate because they can be very subjective to tests. Testers need to be able to go manually into features to actually evaluate the quality. 

“You want to free up testers to go find your edge cases and try to break them,” said Mobile Labs’ McFall. “There are a lot of contextual things automated testing won’t cover.” 

For example, he explained from a user experience perspective, if something like a drop-down menu or search field doesn’t work the way it should, test automation won’t be able to catch that. It will only tell you it is there, but it can’t tell you if it isn’t user friendly. Additionally, things like splash displays that can cover some of the application’s functionality won’t be detected through test automation. Again, it can detect it is there, but it can’t tell if the object is accessible and interactive, McFall explained. 

Additionally, McFall believes it is always good to have a person available to validate the test cases before they are automated just to make sure it is actually worthy of automation. 

Automate feedback.

According to QA Systems’ Davis, there needs to be a way to share information, results and test analysis in a timely manner. 

Even if testing is being done manually, that doesn’t mean the feedback loop can’t be automated. 

Mobile Labs’ McFall explained  as you manually test things like user experience and capture interactions, you can send your findings back to a tool in an automated fashion such as a real user monitoring system or customer experience system.

“I am running through a manual test case, but what is automated is the interaction of the manual verification with something like my ticketing system so that I know in Jira, it is done, passed and automatically sending back logs and other types of information. That to me is how you can have manual testing in a DevOps environment. The feedback loop itself is automated,” said McFall.

Monitor.

There are a number of monitoring tools that can help implement DevOps, according to Broadcom’s Ahmed. For instance, you can track the number of defects getting leaked across the life cycle to see where the leaks are coming from and how to fix them. 

Risks associated with the release can also be looked at from multiple angles: “Even if you develop fast and automate everything, there is still a question on whether you are going to introduce risk or break something when you go into production,” said Ahmed. Monitoring techniques can provide insight into how much test coverage has been achieved, the quality of those tests, if you did the right testing, if dependencies were testing, and if you are able to identify and remove any security challenges. 

Similarly to starting testing earlier in the life cycle, monitoring can also shift left to get developers and testers involved, according to Ahmed. This will help teams be proactive on how the app will behave once it is deployed and what the monitors and alerts actually mean. 

Don’t treat all applications the same.

For instance, mobile devices and applications are different from browsers and the web architecture. Mobile devices such as computers are very fragile, according to McFall, and there are many challenges around it. You need to understand the programmatic pieces around mobile apps as they relate to their infrastructure and environment. 

“People expect the mobile web to be similar to a desk web architecture. Things can look similar, but under the hood they are very different. It is hard to automate that if you are trying to use common frameworks,” said McFall. 

Enable self-service environments.

According to Quali’s Ber Lerner, another main barrier of DevOps is infrastructure provisioning and application provisioning. “Making sure that everyone gets access to the infrastructure and the applications they need, whether it is people trying to run tests or systems that are trying to do automated tasks becomes a big bottleneck,” she explained. In order to overcome this, Ber Lerner explained teams need access to cloud-agnostic environments, which they can get through self service portals APIs or different plugins. “It gives people self-service access to their test DevOps environments while it is still possible for IT Ops to govern the way that it is done. It makes it possible for people to be fast but still be in control.” 

“At the end of the day, it is about breaking down the barriers you have within the four walls of your organization and making sure people have access to the environments they need, systems they need, and tools they need to be successful,” said Mobile Labs’ McFall.  “Let’s make sure we focus on testing the right things, and the more important things.”