Modern software organizations have two core goals: first, be responsive to customers and preemptively fix issues quickly and second, deliver the capabilities customers want, faster and/or better than the competition. If developers and testers are joined at the hip, it’s easier to achieve these goals. With release cycles getting faster all the time and customer demands increasing in lockstep, we’ve had to devise new ways of working. Traditional testing methods, when performed in DevOps teams, may appear to delay releases. Additionally, given the velocity, volume, and churn of code being released, shortcuts in the tests result in too-high defect rates.
These pains have led to continuous testing becoming a fundamental activity in a DevOps practice. Continuous testing relies upon executing automated tests throughout the software delivery pipeline, as soon as developers write new code. But it’s just as much about process change and frequent collaboration between testers, developers and product managers.
Ideally, continuous testing satisfies both of the previously mentioned core goals – increasing customer satisfaction and quickly delivering a high-quality product. We’re already seeing some hard evidence of this bearing fruit. Our company recently surveyed developers and testers in organizations of 50 employees up to 3500; we learned that 92% of them agree that continuous testing helps deliver faster feedback on observed defects, while 84% agree that continuous testing speeds up release cycles.
That’s promising! But here’s the rub. For most organizations, it’s no small thing to go from heavy manual testing at the end of sprints (aka ‘Scrumfall’) to adopting continuous testing. A central reason behind those struggles is that continuous testing requires the automation mindset in a team. That means testers must learn new tools and skill sets. Without test automation, it’s not possible to keep pace with the demands of a high-velocity development organization. Addressing the two primary challenges below is integral to success with continuous testing.
- Helping testers learn how to code
Most manual testers shy away from basic coding skills; therefore, learning how to write and maintain automation scripts can be a struggle. Granted, testers don’t need to write advanced code like a developer; however, at a minimum they should be able to read, write, and modify code associated with the automation scripts. Don’t be fooled by DevOps vendors saying their tools completely eliminate the need for testers to deal with code. In fact, 84% of participants in our survey said that if testers had better coding skills, it would help with continuous testing. Unfortunately, recruiting automation testers with the appropriate skills is difficult – there just aren’t enough of them to go around. Yet 56% of participants said that manual testers were hesitant to learn these new skills. The natural first step to encourage testers is to provide incentives and educational support.
It all starts with changing the mindset around testing. Only 10% of those surveyed say that testers are motivated to learn these new skills. Are they overwhelmed by the technical requirements or don’t see the benefits? The nature of their job is changing and if they don’t adapt, they will be left behind.
Encourage change by introducing automation slowly and iteratively, so that your teammates can realize the benefits on their own. Reinforce the following message: test automation will not replace jobs but provides time to be more creative, search for new problems, and let go of repetitive mundane testing tasks. Develop clear organizational goals for continuous testing and measure them. A majority, 80%, say that having better metrics to track progress would speed up adoption.
- Establishing new processes and workflows
A successful continuous testing implementation is not just about learning how to code. Testers need to learn how to engineer automation scripts the right way from the beginning; otherwise, they may spend too much time maintaining scripts. In our survey, 84% of respondents saw their maintenance efforts increase. And that means they’ll have less time testing new features and code. Automation scripts must be properly engineered from the beginning to prevent testers from spending too much time on maintenance. Instead, their time should be spent testing new features and code changes.
Adapting to changes in application functionality and incorporating dynamic data requires scripts to be written leveraging variables, data lookups, and page objects. For instance, consider a login script. Usernames and passwords are hardcoded but, if those passwords change, a dynamic lookup action knows where passwords are stored and will automatically update the script. That way, testers don’t need to go back and change the script.
Our survey also reported that teams new to continuous testing were not prepared on how to incorporate automated workflows by using the DevOps toolchains, such as Git, Jenkins, Docker, etc. Also, CLIs can be used to speed up manual tasks and integrations between Dev IDEs and tester ALMs. Testers and developers need guidance on how to effectively work together. This reinforces the need for holistic training, not just concerning technical skills but also best practices for processes and communications. The velocity of change makes regular interaction among members critical. Communication barriers between developers and testers have existed for decades; as such, collaboration tools can help cross that barrier, but nothing beats personally interfacing to develop a strong rapport. Frequent collaboration keeps everyone on the same page, with a shared understanding of evolving issues and requirements.
I can’t overemphasize the need for teams to invest in on-going training. Continuous testing benefits organizations not only with speeding up the process and reducing defects but allowing teams to gain better practices in automation, including an understanding of where to use automation and when manual testing is required. It also encourages, if not forces, team members who used to work independently to work together toward a common goal of developing great software that users love. And that is the ultimate goal.
