Software development went and got itself in a big damn hurry. Agile practices combined with continuous integration and delivery practices have dramatically sped up the development life cycle, reducing project sprints from months to weeks to even days.
Testing, on the other hand, is not quick. Manual testing is a time-consuming and labor-intensive process to ensure a piece of software does what it’s supposed to, no matter how fast it was developed. The challenge for both developers and testers in a landscape increasingly dominated by agile is figuring out how to bring testing in line with the pace of development and delivery without sacrificing quality.
“When that bottleneck around deployment was taken away and all of a sudden code could truly flow naturally and smoothly into production, there was a step back in terms of quality,” said Matt Johnston, chief marketing and strategy officer of Applause (formerly uTest). “[Customers] would come to us and say we want to test one build per week. Then all of a sudden between the shift to agile and continuous integration, suddenly they had 10 builds a week. Dev and DevOps could move faster and QA wasn’t able to keep up, so organizations just decided to launch and see how it goes.”
In this shift to continuously integrated and delivered software, in which developers integrate code into a shared repository several times a day (enabling a change or version to be safely deployed at any time), testing is shifting toward a continuous model as well. Through a combination of test automation and the merging of development and testing processes under a Dev/Test philosophy, testing providers and QA teams are beginning to implement practices to test software builds as rapidly as they’re being churned out.
“In today’s world, it’s just too slow to be in this sequential process of dev, test, deploy and manage,” said Tom Lounibos, CEO of SOASTA. “Developers have always tested; they’ve always done unit testing, application development testing, but then they throw it over the wall to the QA guys who would do functional, load and other testing. Testing is now starting to be done by developers far more frequently. QA professionals are still very much there, but they’re trying to automate the process as well. That’s the big shift around continuous integration and continuous testing, so that speed can be improved without rushing software out the door.”
Automating the conveyor belt
Continuous testing doesn’t happen without automation. In traditional manual testing, a developer checks in a change and it goes through a build process that may take hours or days to produce feedback. Automation accelerates the cycle, checking the code and providing feedback in a matter of minutes. By no means does this signal an end to manual testing—which remains essential in exploratory and regression testing at the Web, UI and mobile level—but automation frameworks and tools are proliferating throughout the process.
“If there’s one overarching principle, it’s to automate everything,” said Steve Brodie, CEO of Electric Cloud. “That means you have to automate all the types of testing you’re doing. The key is orchestrating the pipeline: It’s one thing to have these silos of automation doing automated load or regression testing, automating builds or even deployments. But what you need to do is automate that whole end-to-end pipeline, from the time the developer checks the code, all the way through to production and deployment.”
Automation requires much effort on the part of the organization to do correctly. Machines need to be provisioned and configured, manually or virtually, and testing environments need to be spun up and deployed for each application. Yet Brodie believes a misconception exists around automated testing and the quality of those tests.