Agile is all about change. IT leaders adopt agile to accelerate the pace of change for their business-differentiable software. Adopting agile requires changes throughout the people, processes and technologies involved in building that software. Development teams must significantly change their structure, culture, tooling and daily activities for agile. And once agile is adopted, the applications under development change on a daily (or more frequent) basis.

However, amid all this change, one thing tends to remain the same: the software testing process. One recent study reported that 70% of organizations have adopted agile, but only 30% automate testing. A separate study found that while agile adoption is now near 88%, only 26% of agile organizations have broadly adopted test automation. In other words, testing processes remain stuck in the past even as organizations invest considerable time and effort, transforming their development processes to meet today’s and tomorrow’s business demands. Not surprisingly, that same study reports that the majority of these agile teams are not satisfied with the current pace of change.

(Related: How to spread Scrum through the enterprise)

Why does testing lag behind? In most cases, teams want to avoid the perceived pain of transitioning from a manual testing process to an automated one. If the acceleration initiative doesn’t require a change to the testing process, testers typically won’t take it upon themselves to initiate process change. And even if change is mandated, test managers tend to reassure the organization that adding some UI-level test automation to their existing test process will be sufficient.

Any attempt at test automation is definitely a step in the right direction. However, more is needed to meet the needs of modern development processes. Forrester reports:

“Functional testing is one of the most crucial, time-consuming, and expensive steps in Continuous Testing—so it’s necessary to automate this testing, and to automate it at higher levels than most agile teams achieve today… To keep up with the pace of the rest of the pipeline, functional testing needs to be automated and optimized from beginning to end, from the design and automation of test cases, to their execution in the overall testing process, all integrated in within the broader CI/CD automation process.”

To simplify:

Continuous Testing > Test Automation.

Testers across the industry have already recognized that testing must adapt in order for testers to remain relevant in agile and DevOps processes. In a recent survey of software testing professionals (including testers and managers):

  • 91% believe that the role of the tester must be transformed to meet the needs of today’s processes
  • 70% believe that as organizations embrace agile, the role of tester changes significantly
  • 45% increased the level of functional test automation during the past year in response to agile

Moreover, Gartner confirms that as organizations accelerate software delivery, “testing simply cannot keep pace [with demand for faster delivery] due to a heavy reliance on manual testing processes.” Additionally, survey after survey is now citing “inadequate test automation” as a top barrier to agile adoption.

The writing is on the wall: Manual software testing must evolve in response to the shift to agile and DevOps. No matter how many testers you employ, it’s simply not possible for manual testing to provide agile developers immediate feedback on whether any of their constant changes impacted the existing user experience. Without this safety net, agile becomes a tremendous business risk.

As the industry has begun to recognize, test automation will be a critical step in controlling that risk in an accelerated delivery process. However, not all test automation is equally effective for this purpose. When manual testers adopt automation, the most common path is to focus on UI test automation. However, most automated UI testing is not perfectly suited to the type of Continuous Testing that agile and DevOps requires because:

  • Testing begins late in each sprint: Tests usually cannot be implemented until late in each sprint, when the UI and dependent components such as back-end APIs are finally completed and available for testing.
  • Slow execution time: Tests are time-consuming to execute, so it is not practical to run the complete regression test suite on each build. This means the team lacks instant feedback on whether their changes impact the existing user experience.
  • High maintenance: UI tests require considerable rework to keep pace with the frequent changes typical of accelerated release processes. This results in slow, burdensome maintenance and/or causes automation efforts to be abandoned.
  • Frequent failure: Test environment inconsistencies (e.g., intentionally dynamic elements or frequently-changing dependencies) can also cause overly sensitive UI tests to fail—again, requiring burdensome follow-up and/or causing automation efforts to be abandoned.

Automated UI testing certainly is not is dead. However, like everything else, it too needs to be re-engineered for agile and DevOps.