Let’s start off by stating the obvious. Right now, it’s highly likely that your organization is struggling to deliver quality software on time, and on budget. Scope, cost, schedule — you’re told you can have control over two, but you’re likely struggling to nail down even one of the three.
And you aren’t alone. In fact, over 70 percent of IT projects fail. This failure dooms careers and companies, causing a collective drain of over $1.7 trillion on the global economy annually – an eye-popping sum that exceeds the GDP of all but nine countries.
RELATED CONTENT:
Test environment management an obstacle to continuous testing, report finds
Facing the challenges of continuous testing
While we all know that having a continuous testing process in place will dramatically reduce the risk of failure, a lot of organizations still struggle with the adoption of continuous testing methods. So how can you ensure your team wards off failure and avoids becoming another statistic?
Read on to find out how to avoid some of the most common hurdles and pitfalls.
Make proper requirements & planning a top priority
The single biggest reason projects fail is a lack of actionable requirements.
Right now, it’s likely a huge number of your requirement tickets only consist of a short summary sentence in the title. Or maybe someone has attached a monolithic PowerPoint presentation to a single ticket, documenting new features across multiple sprints. Perhaps instead you’ve got a few thousand auto-generated defects from an accessibility audit that was completed two years ago? It doesn’t matter. Regardless of what format they take, inconsistent requirements force developers to rely heavily on their own subjective interpretations of what to build. And this eats into their productivity, and significantly decreases your chances of success.
How do I get it right?
To be successful, requirements need to be clear, actionable, and testable. This can help you avoid:
- Developers building something other than what was intended,
- time wasted on meetings to align on requirements,
- dramatically slower production times,
- missed schedules due to having to rebuild during UAT, and
- an increase in problems making it through to your customers in production.
To reduce risk, keep your requirements as minimalistic as possible and build towards MVP. And at an absolute minimum, ensure you have a user story, acceptance criteria and a testing plan to keep you on track.
The goal of requirements should be to have enough information in every ticket for any developer to pick it up and run with it, without having to spend time deciphering email threads or setting up meetings to understand the ask. Having a good set of requirements templates will help, along with having a unified project-wide guide documenting workflows, so everyone involved knows how your process works.
Refine requirements before you build. All new requirements should be reviewed by your architects, developers (ideally the same ones who will be doing the work), and your quality assurance teams. Make sure to allocate time for them to ask questions and give their feedback. It’s also important for someone from your QA team to be engaged during planning to help create testing plans, write test scripts and identify test data sets needed. Managers and sales teams simply passing on requirements to developers without making themselves available for discussion, or encouraging developers to suggest improvements is a horribly outdated way to work (and almost guaranteed to end in failure).
Create a unified source of truth for your requirements. To achieve company-wide alignment on what “done” means, and the process for how to get there, make sure every team touching the code works with it in the same way. Far too often “build squads” and “maintenance squads” each have their own way of doing things, their own tools for requirements or defect tracking or their own deployment work-flows. This inconsistency will bring chaos to your project.
Get everyone on the same page. Consider giving everyone in your organization access and training on your requirements planning system. Insights can come from anywhere, and keeping systems transparent and open can help you get important information faster. When the right people have the means to communicate directly, it removes the unnecessary time and risk associated with a complicated chain of command.
Create consistent issue templates. Once everyone is trained on the process and tools (make sure to build this into your on-boarding process!) you can create consistent issue templates. Start simple, with “new feature”, “enhancement”, and “defect”. It’s likely some issues won’t have every single necessary detail, so developers can be encouraged to enhance issues with their assumptions before coding. However, make sure everyone knows your “getting to done” process, and make sure it’s culturally acceptable in your organization for people to send tickets that don’t meet standards back to the creator for more details.
Always ask for feedback. Every team will have slightly different needs and requirements, so it’s important to get feedback on your templates. Make sure you’re documenting what everyone needs, and if your software has to go through accessibility testing, security testing or any other compliance testing, make sure those requirements are stated up front so everyone can build with those standards in mind. Likewise, make sure your coding standards, approved licenses and code review processes are publicized and kept up to date.
Now that you have a set of requirements templates that your team is happy with, set up a process for the development lifecycle. We’re all familiar with the kanban methodology columns – “To-Do”, “In Progress”, “In Review” and “Done” – so consider adding a “Planning” column to ahead of this and mandating that nothing moves into “To-Do” until it’s been reviewed and everyone on the team is happy with it. This will keep your “In Progress” tickets out of the “To Do” pipeline and let everyone know that the to-do list is actually ready to be worked on.
Automate provisioning, builds, and data generation to automate testing
Continuous delivery requires automation as you move code between environments. It’s important to work out a plan with your DevOps team to ensure environments can be automatically provisioned and code deployed to them for testing. While the industry on the whole is improving, far too many enterprise-grade companies still have a manual build process – and while being able to make “daily builds” was great 20 years ago, the modern standard should be more aligned with, “Can you create a build on-demand, with one command?”
These builds don’t just require an environment and code. They also need to be tested at every step, to help you catch problems early and at the source. Testing requires access to quality test data, and far too often, tests are run on small data sets that only test perfect-path results.
Expect the unexpected
Perfect path testing leaves a lot to chance. What happens when an API you depend on fails? If you don’t know, or didn’t build in the right fallback or user messaging, you’re going to have unhappy customers. So make sure you have a process for creating comprehensive and realistic test data, or pulling sanitized data from production that you can use as part of testing. Having realistic tests, imperfect-path tests, or even some chaos testing, can really help you at this stage.
Also bear in mind that modern systems are complicated and often interconnected. We are not just building one stand-alone application; we are building something that relies on numerous back-end systems and 3rd party integrations, which may or may not come with a sandbox mode for testing. This means that in addition to “internal” test data, you may also need simulated virtual or “mock” services for those times you can’t test against production or a 3rd party service.
By introducing mock services, you can write unit tests that perform negative testing or chaos testing, ensuring you know how your code will perform under a variety of real-world circumstances. Sounds complicated? It’s not if you use the right tools. And if your organization is struggling with unit test coverage, focus on API-endpoint testing as a starting point.
Reduce, reuse, recycle
While automation can help, another thing your team can do is optimize and reuse code in order to reduce overall workload. This decreases the number of tests required and the effort needed to fix bugs and make enhancements.
Look to common areas – like user sign in pages or managing profile forms – and ensure you’re reusing validation and error handling. This will help give a consistent user experience, as well as simplify the amount of code you have to support. Given that 70% of testing is still manual, small steps like this can help free up your QA engineers, giving them more time to work on automation tasks.
Why automate?
Allocating time to create automated tests should be high priority. Once you create a test, you only have to touch it again if the underlying code it’s testing has changed. Therefore, it is endlessly reusable with a tremendous ROI. The effort savings will typically pay off within one sprint – imagine the savings you will get over a year, or the lifespan of that code.
Automation is not just about time savings; it also significantly increases quality. Even a simple dependency update can have a knock-on effect on other parts of the system, with potentially sweeping ramifications. The more you automate, the easier it is to do full-system tests to ensure your “simple change” has the intended outcome (with no unwelcome extras).
It gets even better – many modern tools allow you to record front-end UI tests, and replay them, meaning with the right tools it’s a zero-cost exercise to build the infinitely reusable automated tests vs. just doing manual testing one time. That’s a no brainer.
Work smarter, not harder
Assuming you have clean requirements, another huge win is the ability to automatically generate your test cases. Manually building and managing test cases is a massive time sink – and therefore it seldom gets done thoroughly. However, with the right tools and requirements, you can shrink the effort down a size or two.
During your automation transition, you’ll likely experiment with a number of tools, and the cost of open-source tools makes them appealing. But when you’re ready, look for enterprise tools that will enable you to import any tests you created. Being able to reuse things you’ve already got on hand will make it much easier to keep momentum.
Don’t feel like building everything yourself from scratch? Many frameworks and platforms have great tools for sanity checking their own best practices or a predefined set of functional tests. While these tools won’t always be tailored for your specific needs, some test coverage is better than none. For web projects, Lighthouse is a great tool that can be easily be added to your build process to check web pages and report on general performance, security, or usability issues.
Bring tools to developers
During development, the cost of fixing a bug is dramatically cheaper than trying to fix it once it goes into production. A stitch in time saves nine (or in this case 1000 – yep, by some accounts it can cost 1,000 times more). Recent failures by Boeing and Samsung highlight the damage a lack of proper testing can have on your company’s reputation.
It’s not hard to see how these sorts of issues happen. With the old way of doing things in linear siloed phases, testing wouldn’t happen until developers were finished. Given that testing was typically time-boxed, it was rare all the tests would run; then on top of that, the resulting data may not make it back to developers until the following sprint. These delays introduce a high level of risk, with plenty of time for the original developer to forget context or for the project to have been handed over completely to somebody with no prior knowledge of the requirements.
Instead, a quick win is finding tools that easily integrate with your developers IDEs, or can be run from the command line or in code. By integrating all the tools together, you remove the hurdle of multiple systems and environment variables, making it easy for developers to find bugs in their own code and immediately fix them. To reap these benefits however, it’s important that everyone writing code has access to the testing tools, knows how to use them, and will use them.
Build a collaborative culture of rapid feedback
The goal of continuous testing is to create rapid feedback loops, but that’s not limited to code. Since software development does not end at launch, it is important to cultivate a culture of continuous feedback; and opening your development stand-ups to anyone who has work slated that sprint will ensure your team is better able to get the answers they need. Having cross-functional retrospectives will also help ensure your continuous testing initiatives are working for everyone, and will give you the opportunity to fine tune as you go. If you can’t find a time for everyone to attend, offer the ability to submit feedback via a “suggestions box” (you can easily set this up with a Google Form).
Remember, retrospectives are not just for calling out things that need to be improved – they’re also a great opportunity to call out the people who are doing things right. Teamwork is key, and nothing builds rapport like gratitude. Things will not always run smoothly, but one key metric to follow is how fast you can address issues that are causing pain – and the easiest way to do this is to capture the mood during a retrospective.
Adjusting to the new normal
Even with support from all stakeholders, simply mandating that everyone switch to use continuous testing methodology without training and time will cause your initiatives to fail. Continuous testing cannot be mandated to just one team, or by one team. It is a fundamental commitment to quality that must be embraced by everyone in the organization. Your teams will need time to adjust, adapt to new processes, learn new tools, and ask questions of one another – and all of this takes time.
Since most existing automated tests and testing data tends to reside with quality assurance teams, it is great to lean on them to champion change and share their testing knowledge across your organization. Quality assurance testers and engineers will also have a lot to offer and, just as we have agile coaches, can function as continuous testing coaches throughout the organization.
With all the required training, you may see an initial slowdown of productivity during the transition. It is important not to panic. A lot of things contribute to story point decline, but it’s important to let devs track non-development tasks – like meetings, training sessions, and retrospectives – to the story point count so you are tracking an accurate picture of how much the team is getting done. Change will happen fast, but it will not happen overnight. In the meantime, it’s important for everyone to stay flexible.
Conclusion
The best way to ensure your continuous testing initiatives are successful is for your organization is to:
- Be proactive with requirements,
- work to share knowledge and access across teams,
- automate as much as you can (and make sure all teams can reuse the work)
- generate appropriately robust test data, and
- be kind to your future selves by budgeting time for continuous feedback and improvement.
