Agile is no longer a long sought-after dream for businesses. It is no longer a secret weapon for organizations that wish to stay ahead of the curve. Agile, for the most part, is the modern way of working. By now, most organizations know the purpose of agile: to build higher-quality software faster. But in order to ensure that quality, businesses have to make sure their testing processes can keep up with this new pace of working.
Without testing, agile initiatives will never be truly complete or succeed, according to Eric Taylor, director of agile software delivery for AgileTrailblazers, an agile consultant firm. “With the speed that agile brings, the only way to really achieve that speed is to actually get your testing in order,” he said. “You have to really put testing first, and change testing to be the mindset of the entire organization as opposed to being this thing you do at the end.”
Organizations who don’t address testing upfront when they go through an agile transformation will experience bottlenecks in their processes because testing needs to keep up with the pace of development and the rate of change. Therefore testing shouldn’t and can’t be an afterthought, according to Zubin Irani, CEO of cPrime, an agile service company. “It is a very difficult transition to make, and it is a very painful one that often doesn’t get done until later. Usually what happens is you go through a transformation, you realize it isn’t working and then finally you address testing.”
(Related: Driving a digital transformation)
Since this is a complex transition, many organizations fall into the trap of pushing it to the side. But according to Wayne Ariola, chief strategy officer of Parasoft, they will soon realize testing can’t be ignored.
“Unfortunately, any kind of process impact or technical initiative that businesses have engaged in, the last group to truly be considered associated with the transformation is test,” he said. “But if you look at agile processes, the biggest impact associated with the change is testing.”
And it is not only agile pushing the importance of testing. We are in a new world where businesses are now digital, and software is your business. “I believe that most companies look at testing as a necessary evil more than a value-add activity,” said Thomas Murphy, research director at Gartner. “Agile ties into this because generally if you are going to survive in the digital world, you must utilize agile and DevOps practices to achieve the pace and maintain the quality that is required to remain agile as a business.”
Testing in an agile world
No longer do testers have the luxury of waiting until the end of the cycle and taking their time to test the code. “Testing is less a phase and more a set of activities now,” said Ian McLeod, chief product officer of Rogue Wave.
The entire team and life cycle has to become “test infected” in order to ensure the success of agile, according to AgileTrailblazers’ Taylor.
“You can’t really have these periods of introducing a lot of bugs and going back and fixing them,” he said. “The role of the tester really being on the team is sort of the guardian of quality, and their job is to make sure the quality of the work everyone is doing is staying high and really enabling the developers, product owners [and] Scrum master to all care as much as they do about quality.”
In order to succeed, teams should:
Devise a plan. “This is the standard pablum…start small, establish your goals/objectives first, assess your current position and challenges,” said Murphy. “It depends on the starting point. Make sure you see quality not as someone else’s job but everyone on the team.”
To do this, there should be open communication between the business, development and quality assurance, according to Brad Stoner, senior sales engineer at AppDynamics. “If an organization wants to succeed, they have to understand that they are going to change the process from front to back, so business to development to QA,” he said. “There has to be a strong feedback loop and QA has to be empowered to be able to say, unless we make this app testable, we are always going to be your bottleneck.”
Stop asking “Are we done testing?” That question is more associated with a sprint and scheduling or a burndown perspective, instead of being associated with the business, according to Parasoft’s Ariola. Instead, teams should be asking, “Is the risk of the release candidate acceptable?”
“This is a massive change associated with how someone approaches the whole concept of quality,” said Ariola. “When we talk about risk, we talk about if we are mitigating the true business risks associated with the organization.”
Automate. Automate simple tasks that can easily tell you if the release is performing as expected. This will help QA focus on good builds and getting instant feedback to development if there is a problem, according to AppDynamics’ Stoner. “Instead of having QA look at these bad releases multiple times a day, they don’t have to waste their time,” he said. “This is the first way to accelerate the process, and it is very low cost and easy to implement.”
To get started, testers can automate test environments, leverage cloud environments, automate E2E scenarios, monitor app performance, and leverage scripting skills, according to Anand Kamat, group program manager at Microsoft.
Rogue Wave’s McLeod believes it is easier to do automation if you are starting from a blank slate rather than converting your process. For organizations that decide to convert, they need to invest in quite a few sprints, he explained.
According to Kamat, there will still be a need to manually test. For instance, teams will still need to do a bug bash against the solution; product managers will have to play with the product; and early adopters will use the solution in real-world scenarios.
Kamat explains that in order to get a full comprehensive view about the quality of a product, automation test metrics, manual testing, exploratory testing, user acceptance testing and testing in the real world are all essential.
Exploratory test. One of the common pitfalls of automating your tests is thinking that because 100% of the automation tests pass, that guarantees 100% of the use cases, according to Kamat. “That can give you a false confidence,” he said. In order to avoid this, testers should also be performing exploratory sprints. “Exploratory testing is really having subject matter experts test end-to-end scenarios. They will test for user experience, test for usability, and they need to provide quick feedback,” he said.
AgileTrailblazers’ Taylor adds that exploratory testing is one of the most important parts of testing because “It gives an analytical individual the time to go through the site and think of all the kinds of odd scenarios that others might not have been thought of.”
Unit test. Unit testing must be in place in order to validate user stories as fast as possible and understand the impact of changes, according to Parasoft’s Ariola. “Without that baseline level of visibility, you are basically going to defer a lot of quality tasks to much more expensive times, and it is going to lead to failure or outages,” he said.
This helps in reaching a test-first design, according to Rogue Wave’s McLeod. “You don’t write code unless you have a unit test to test it, or in some cases you don’t write code until you have a unit test to test it,” he said. “I write the test for my code first, then I write the code to make sure it passes.”
Unit tests should include a range of inputs and outputs, the functional requirements of the component you are testing, all the false or false conditions that may be encountered, and the full range of data, according to McLeod. “It is not unlike writing a spec for the component; it is just the spec is in the form of tests. Rather than having a spec, you are writing a test saying here is the test to make sure this component does everything it is suppose to, then you write the component to pass the test.”
Perform static code analysis. Quality is the No. 1 priority of software, and it needs to be looked at early in the process, according to McLeod. “Quality is cheaper if you do it early in the process because you can make sure your stuff is right the first time you check it in, and it doesn’t break the build.” One way to do this is to perform static code analysis on your software. Static code analysis gives teams an overview of vulnerabilities, defects, security weaknesses, and code errors so they can fix problems faster.
Test-driven development. TDD is the foundation of good development, testing and writing unit tests, according to cPrime’s Irani, but it can also be one of the most difficult things for a tester to learn. Being able to perform TDD takes a level of maturity, but when it is performed correctly it can be very powerful.
“It puts quality first. It is just prioritizing quality, and allowing you to think about it in a different way because you had to think about it from a testing perspective, which is more of a user perspective. You tend to have better-quality stories that are a little more thought out ahead of time so you go into your development with a better mindset and a view of what you are building,” he said.
Have independent test groups. Having independent test team members acting as full-fledged members of the development team while maintaining independence will help organize and manage the development team, according to Rex Black, president of consulting firm RBCS.
This should not be confused with keeping testers as a separate team, according to AgileTrailblazers’ Taylor. Testers need to be fully committed to a given team.
In addition, there should be a separate group as part of the independent test team responsible for building and maintaining a regression test suite over a long period of time, Black explained. “Have an outside group look at things like usability, security, performance, and reliability because those are all very difficult to integrate into sprint teams especially if you are dealing with scale. Properties like performance, reliability and usability are emergent behaviors of the system as a whole as opposed to behaviors associated with a single feature or user story. It makes sense for those test task to be taken outside the scope of development teams.”
Focus on the testing pyramid. According to Taylor, the pyramid puts unit tests at the bottom, integration tests in the middle, and user interface tests at the top. The hardest automation tests to write are for your UI because they change the most and can be pretty complicated. Unit tests are easiest to write. “Put most of your investment in unit tests and integration tests, then just a little in UI because it is brittle, hard to manage and hard to maintain. You need some, and you can build over time, but start small,” he said.
Microsoft’s Kamat adds that having a service or API testing strategy with exploratory testing is a better alternative to UI automation.
Utilize beta groups. One practice testers can use to ensure their product is working without sending it into production is to have it tested internally to a limited amount of users. “What that allows you to do is get field testing and real-world testing back so the developers can react quickly and then fix defects, bugs or any kind of confusion that occurred from that testing,” said John Basso, CIO and cofounder of Amadeus.
Additionally, instead of rolling out to 100% of your users, teams can roll out to only 10% so that if anything happens it only impacts a subset of the population and can be fixed and re-deployed quickly, Basso explained.
Test less, but fix faster. Since testers no longer have the luxury of waiting until the end of a development cycle to test, Rogue Wave’s McLeod believes it is sometimes easier to test less, and then live test. Of course, he notes that it all depends on if your software is mission-critical. “If your software keeps airplanes in the air, it is probably not the best approach,” he said. “But if your software is hosted, fast, easy to update, and maybe isn’t too mission-critical, I think to some degree you should take a little more risk in your release. And since you are working in smaller increments you can fix it much more quickly.” The idea is to deliver value to your customer and to be in a position where you can respond if something doesn’t work, McLeod explained.
This is taking a shift-right approach rather than a shift-left approach, according to Microsoft’s Kamat. “If we can’t do enough testing before you release, you have to complement the risk exposure by monitoring the application in production, so that if there are any issues, you are the first one to detect them, and you can rectify them quickly,” he said.
It is important to note that this does not eliminate the need to shift left, Kamat explains. “The more testing you can do early in the life cycle with more automation, the better the results,” he said.
A lesson from Amazon Prime Day
It is one thing to talk about the importance of testing and quality, but it is another to experience it out in the real world and suffer consequences because of it. Having a software failure, bug or defect released into production can erode a business’ brand, according to Parasoft’s Ariola.
Amazon recently held Amazon Prime Day, its biggest online shopping event. Since the company holds these lightning deals once a year they had plenty of time to prepare and plan, but yet even the biggest technology companies are capable of making mistakes. The company ended up suffering from a software glitch that prevented customers from being able to put items into their shopping carts, making customers very frustrated.
Users took to Twitter to voice their complaints with the hashtag #PrimeDayFail.
Twitter user @pcg79 wrote: I removed my two #PrimeDay items from cart. Now I can’t re-add them. “You have claimed this deal”. Get it together, @Amazon. #PrimeDayFail.
While @PreroDesign wrote: #PrimeDay is giving me nothing but add-to-cart fails. Nice job @amazon. Another #PrimeDayFail.
According to Ariola, the lesson to take away from Amazon is that software quality is no longer optional. The cost of quality and the cost of failure continue to rise.
“It is funny to see things that I would consider or the industry would consider to be a minor software issue just blow up in social media. It is a good example of how social media and how the general public observes and who has the right to criticize the quality of software,” he said.
Agile doesn’t mean fail
There is a popular saying in agile and general software development initiatives: Fail fast and fail often. The point of this saying is that organizations will fail, but they will learn from their failures. While this saying might work for some companies, this is not a good mindset to have.
According to RBCS’s Black, falling into a fail fast, fail often mentality is dangerous because it makes it sound like getting things right the first time doesn’t matter. “I suppose this attitude is appropriate or at least tolerable if you are building the Kim Kardashian game to run on someone’s mobile phone,” he said. “It is not an attitude that is in any sense tolerable if you are building self-driving cars.”
Black explained organizations often adopt the philosophy of just throwing stuff out there because if it is broken they can just release another update, but that can make the life of testers and developers difficult. “As software becomes more and more a critical part of the infrastructure of our lives, the importance of quality in software goes up, not down. Therefore sloppy and ‘ready, aim, fire’ approaches become less appropriate rather than more,” he said.
Instead of rushing to get stuff out there quickly, developers and testers should ask themselves if they can rush their product out because it doesn’t matter if it’s broken, or is it something that is being implemented in self-driving cars, elevators, pacemakers, and other critical solutions, Black explained.
“Agile isn’t just stuff that gets out fast. The compromise that is going to be made is almost inevitably quality because it is certainly less visible than budget and schedule, and people are very good at counting days and counting dollars. But quantifying what is an acceptable level of quality, organizations aren’t always good at that, and that can lead to some bad decision-making,” he said.
Updating your skillsets to stay relevant
Understanding that addressing test is one of the most essential parts of the transformation is only the first step. Businesses need to figure out in this new agile world who is responsible for testing and what constitutes testing, according to Gartner’s Murphy. “In an agile organization, everyone plays a role in quality and testing. Developers test, testers test, product owners test, and, yes, the users do certain types of testing,” he said.
So when developers and testers are stuck in the middle of this transition, they need to understand their skillsets need to change immediately.
According to Parasoft’s Ariola, testers need to have a clear understanding of the business and how software is going to impact the business. “This means your blinders associated with just writing tests need to be removed, and you are going to have to understand a much more global perspective of what you are doing as a tester,” he said.
Instead of just testing parts of the application, testers also need to understand its breadth. “You have to understand how your application runs, but you are also going to have to understand the dependencies associated with the application,” said Ariola. “Meaning, what does it connect to? Are there third-party connections from web services? Are there other APIs that this application is dependent on in order to do its job? Does it connect into a mainframe? You are going to have to get to know the architecture associated with that application quite well.”
Having a thorough understanding will also help testers monitor and react better. “They need to be able to really understand how an application works so when it fails they can better make sure the quality is there because they understand how it works, and understand how it could break,” said cPrime’s Irani.
The expectation is that testers are going to be able to participate in not only the discussion about requirements raised as user stories, but also discussions about automation of tests associated with behavior-driven development and acceptance-test-driven development, according to RBCS’s Black.
The job of a tester is no longer to figure out how to manually do things. Their job is to ensure the quality of a product, according to Amadeus’ Basso. “It becomes less about the testing and more about how you can make this whole ecosystem of development, deployment and monitoring better in such a way you can make the quality higher,” he said.
This new role is being seen as a quality engineer, according to AppDynamics’ Stoner. “Quality engineers have to have some coding skills so they can sit with development and build test cases side by side,” he said. “Quality engineering is increasing the quality of builds and developments so that less time has to be used finding these issues. To do this, testers need to get brushed up on their automation skills, Continuous Integration skills and Continuous Delivery platforms”
In order to brush up on those automation skills, testers need to become more technical. This means being able to write code and scripts in order to automate things. “Testers need a growing amount of ability to write code because most testing tools and the most effective way with most systems to automate is you must write code, you must understand the code you are testing, and you must feel comfortable treating your testing assets as code elements,” said Gartner’s Murphy.
According to Trailblazers’ Taylor, it doesn’t necessarily have to be any one language in particular. “What I look for is whether or not they have the ability to pick up some form of automated test scripting, which is really a lighter form of programming,” he said.
For organizations that have hundreds of manual testers, they can’t just hire someone to automate all their manual scripts, according to cPrime’s Irani. “You have to spend the time and money to invest in architecting out how you should be automating testing,” he explained. “You have to think about who you have that can learn those skills, and then acquire the gaps of skills you don’t have.”
In addition, you can’t just train all your manual testers to become automation test engineers. “You have to architect a solution, figure out which of your people can make the transition, and then provide them with the training and skills they need,” said Irani.
On the other side, as roles and responsibilities between testers and developers become more blended, developers need to be more familiar with tests. They have to understand the value of a good unit test. “Writing good unit tests around your code gives you a safety net to make changes down the road,” said Taylor. “If you want to continue to go fast, you need to have that safety net so you can make changes without fear or without having to second-guess yourself, or without having to introduce a lot of defects into production.”
And as developers become more “test infected,” the importance of testers becomes greater. “Developers are typically more familiar with code and writing the unit test, whereas testers have an eye for details, they have expertise in terms of connecting with real-world customers, and they have a better understanding of application topologies in the real world.”
In order to work in this new world seamlessly, developers and testers need to make sure they have good communication skills in place because they need to be able to work together to find and execute the right suite of tests. “The more testers understand, the more respect they are going to get from developers,” said Irani. “The more respect they get from developers, the more they are going to work with testers, collaborate and teach them as well to help them get better.”
Stakeholders need to change their mindset
Transitioning to agile is not only an initiative that happens at the team level. Stakeholders and executives need to be able to change their way of thinking to accommodate this way of working, according to Microsoft’s Kamat.
According to him, stakeholders have grown up with a mindset that if they feel confident about releasing a particular solution, they should release it. “Stakeholders need to understand that just by throwing more people and more hardware doesn’t mean they will be successful,” he said. “The agile principles don’t say that you are still going to do everything you have done before, but in a compressed timeframe.”
The intent is to focus on what has changed, what is being used in production, use the data points to test, and have confidence in the testers that they understand what the business risks are.
“Testers in an agile world need to be supported by the stakeholders to say yes, we are going to understand the risk, and we are going to mitigate the risk by monitoring the application in production.”
