“You’re only as fast as the slowest element, the slowest phase in your software delivery pipeline,” he said. “You’ve got to automate the QA environment and the systems integration testing environment. You’ve got to deploy into the performance-testing environment. If you have user acceptance tests, you’ve got to deploy it there too. But a lot of people think that by deploying faster with Continuous Delivery, quality will suffer. But what’s fascinating is that the inverse is often true, particularly if people are releasing more quickly because the batch size is changing. The magnitude of changes you’re deploying is much smaller.”
Once deployed, an automation solution is not without its kinks. Automated tools are still relatively new, and the process can result in inconsistent reporting, false positives and botched execution. Hung Nguyen, CEO of LogiGear, said the idea is there, but the biggest challenge to automation is smoothing out the release process.
“Think of your entire development cycle as an automated assembly line,” he said. “Once you turn on the conveyor belt, you don’t have to worry about what’s going to come out the other side. But when you get to the system level of testing, thousands and thousands of test cases are running against these virtual machines, and it tends to have some timing problems. Open-source and commercial tools used in combination are just not robust enough yet, so you end up with a lot of so-called false positives and end up debugging.”
The rise of Dev/Testers
The ripple effects of Continuous Delivery are changing the way developers and testers work together, blurring the lines between the two roles and skill sets. As a consequence, developers are learning how to test, and testers are becoming entrenched in the development process. It’s the manifestation of a Dev/Test philosophy.
“Testers are moving in closer to the development side, embedded into teams with developers,” said Tim Hinds, product marketing manager at Neotys. “You see this a lot in Scrum and other agile development teams. The testers are well informed about what PaaS developers are working on proactively to design their test scripts accordingly. So that whenever the code has been written and needs to be tested, they’re familiar with what’s occurring and not just getting something thrown over the wall to them.”
Dev/Testers are upending the way organizations approach testing while adopting agile and Continuous Delivery practices. Organizations that in the past have invested in independent centers of excellence for testing best practices are transitioning to have testing resources sitting alongside development resources, and as a result, the role and skill sets of testers need to evolve to fill that Dev/Tester role of ensuring code quality within an agile team.
“There’s a lot more of a need for testers to understand the application architecture, understand the APIs,” said Kelly Emo, director of applications product marketing HP Software. “If you’re doing API testing, you’ve got to understand that API and that programming model, the underlying architecture. You may need to understand its interdependency with other components of that composite app.
“There’s this new hierarchy of testing, where you have testers sitting alongside developers doing more API testing or functional testing at the application level. Downstream you’re still going to have testers managing the regression sets or doing exploratory testing. They can be more of what people think of traditionally as a black box or manual tester. Those rules still exist, but now you have both.”#!A tester’s skepticism
In the shuffle of bringing testing up to speed with development, there is a danger of losing sight of what testing was originally intended to do. Magdy Hanna, CEO of Rommana Software and chairman of the International Institute for Software Testing, implored organizations to not so easily dismiss manual testing or discount the importance of practices such as regression testing in the rush to deliver software.
“With agile, Continuous Delivery and continuous integration, I get very concerned about overlooking the value of regression testing, which guarantees that things work,” he said. “Some projects and teams thought continuous integration would be a good way to eliminate or at least minimize their regression testing, which is always a stumbling block. Sometimes, in order to push the release to production faster, we overlook or undermine the value of regression testing. I’ve seen projects that actually deliver software faster by cutting down on how much final system, acceptance and regression testing they do.