Continuous testing (CT) has been the missing piece of the Continuous Integration/Continuous Delivery (CI/CD) movement, but that’s changing out of necessity. While organizations say they’re “delivering value to customers” at an ever-accelerating pace, customers may disagree with the definition of “value” when software quality doesn’t keep pace. In fact, software quality may actually decrease as delivery cycles shorten.

RELATED CONTENT: Continuous testing at every step
Why continuous testing is so important

There are good reasons why more organizations start with minimum viable products (MVPs) and enhance them over time. For one thing, the dizzying pace of business and technology innovation means that companies must get to market fast. Unlike yesteryear, there just isn’t time to strive for perfection over a long period of time without running the risk of missing the target completely. Speed has become a competitive advantage, but not at the expense of software quality.

One school of thought is, “If my users know I’m delivering 100 updates a week, they’ll be less forgiving of that which doesn’t work the first time.” Tell that to the irate customer who can’t get loyalty card benefits, play a game or complete an ecommerce purchase. Granted, the fix may be delivered swiftly, albeit perhaps not swiftly enough to prevent customer churn because there are always other alternatives and better experiences to be had.

Interestingly, some cultural issues are contributing to the problem. Namely, how testers are viewed and who’s responsible for testing what.

Arguably, everyone is responsible for quality in an Agile, DevOps and CI/CD scenario and yet the effects of an accelerated delivery cycle without an equal focus on quality means that buggy software is released faster.

“When you start to think of testing as a discipline, rather than a role, everyone realizes that testing is part of their role even if testing is not in their title,” said Sean Kenefick, VP, analyst at Gartner. “That’s a big cultural change and it’s been a problem for a lot of organizations. When you define continuous testing, you should define it as four things: test early, test fast, test often, test after.”

Making testers first-class citizens
CT isn’t going to deliver the intended results if testers are viewed and treated as second-class citizens. Historically, the view has been smart people code and other people test, which has never been a productive mindset and is antiquated in light of Agile, DevOps and CI/CD.

“Testing is a noble profession. It has to be valued, which has gone by the wayside,” said Theresa Lanowitz, founder and head analyst at market research firm Voke. “Once Mercury Interactive went under, there was no vendor standing up for testers saying they’re a valuable part of the software engineering process.”

According to Rex Black, president of testing training and consulting firm RBCS, some organizations have tried to get away from having any sort of independent testing team by having developers automate tests.

“That’s something that often sounds better in theory than it is in practice, especially if the developers who are to go and create automated tests are only trained in how to use tools and they don’t get any training in how to design tests,” said Black.

Meanwhile, testers have been told that they need to code.

“The industry has said to testers, ‘If you still want to have any relevance, you have to become more technical, you have to code in Selenium,” said Voke’s Lanowitz. “Testers are writing Selenium scripts that are difficult to write, hard to maintain, brittle, bloated and they’re still doing most of the testing at the interface level, so there’s not a healthy mix of tests and we’ve largely ignored the non-functional requirements of performance and security.”

Fundamentally, organizations should have a philosophy about quality and satisfying customers that not only aligns with their customers’ expectations but is also in line with the brand image they’re attempting to create or maintain. They should look to testers to lead the CT effort.

“So many organizations say we’re getting rid of our testers or our testers aren’t playing as much of a role as they have before. Quality should be front and center, but we’ve moved away from it, but I think we’re moving back to it because software has become so defect-riddled over the past several years,” said Lanowitz. “You need to have healthy respect for the people that do testing. Quality has to be built in and you have to make it everyone’s job.”

The shift-left dilemma
The “test early and often” mantra predates the turn of the millennium. Since that time, the shift-left movement emerged which encourages even more types of testing earlier based on the same, old reasoning which is that it’s easier and cheaper to keep bugs out of the later stages of the software development lifecycle (SDLC). More fundamentally, software quality matters.

Web and mobile development stressed the need to focus on user experience (UX), which hinges on software quality. However, the focus of UX quality can focus too much on the UI level when other types of testing are also important.

The belief now is that developers should do more than just unit testing including API, performance, and security testing. They may also be the ones writing test automation scripts if testers haven’t learned to write Selenium code, for example. However, the shift left hype is rosier than the reality because many developers aren’t even doing unit testing – so how can they be expected to do other types of testing?

According to Diego Lo Giudice, VP and principal analyst at Forrester Research, only 53 percent of developers are doing unit testing today. Voke’s numbers are even more sobering at 67 percent.

“When defects are not remediated properly, that costs you time, money, brand issues and customers,” said Voke’s Lanowitz. “Defects are leaking from sprint to sprint and phase to phase.”

Forrester’s Lo Giudice said developers are spending about one hour per day testing, which isn’t enough.

“You need to give them time,” said Lo Giudice. “If you’re doing unit testing, development time almost doubles. The only way to realize what’s going on is to start using tools that make that visible.”

Forrester recognizes three different types of testers: business testers, technical testers and developers. Business testers tend to focus on user acceptance testing and they have the fewest number of tools available. Technical testers use high-level scripting languages or graphical tools and do the lion’s share of functional testing. They don’t necessarily write Java code to automate tests, which the developers might be doing at the functional testing level. Performance tests have been the domain of performance engineers.

“This is a joint venture among product owners, business testers, technical testers, and developers,” said Lo Giudice. “How responsive the UI needs to be is a business requirement. The product owner is incented to give feedback on the performance he’s requiring. The testers might help build a baseline or interact with a performance engineering test center to identify issues that if not tested early on become very costly to do when there is end-to-end performance testing.”

Providing a great user experience isn’t just about high performance and slick UIs. Security testing should also be table stakes and not just in regulated industries, but there’s more to it than application security testing.

“If an organization approaches security purely from an application security point of view, they’re going to fail because that misses two other important elements which are service security and infrastructure/network security,” said RBCS’ Black. “You have to have what security professionals call ‘defense in depth’ which means hardening all the different layers.”

Then there are microservices applications and other loosely-coupled architectures that require testing at more than one level.

“When you’re dealing with a lot of contracts – interface contracts, service levels, authentications – you have to make sure your object works in isolation with the contracts but somebody somewhere has to have the bigger picture [of] business processes that align a lot of different services made by a lot of different teams,” said Gartner’s Kenefick.

Developers are the first line of defense against bugs, in other words.

The impact of shift left on testers
Assuming more types of testing are shifting left in an organization, where does that leave testers? The division of labor is obvious in a waterfall scenario in which developers throw code over the wall and testers find bugs. With shift left, the division of labor evolves.

“Shift left means making sure that you not only have ample and powerful upstream left-hand side defect filters but also good downstream right-hand defect filters,” said RBCS’s Black. “You’ve got to have proper testing at the system level, proper regression testing and system integration testing, if appropriate.”

Shifting left shouldn’t diminish the value of a tester, it should elevate it in two ways: by reducing the number of bugs that flow from a developer’s desk and enabling testers to spend more time improving testing-related processes.

“As a tester, I might make the rest of the team more sensitive to what might need to be tested and why. I still need that skill, that mindset, that approach, the curiosity, and that gut feel of where a problem might be hiding or a technical issue might be hiding and discovering it,” said Forrester’s Lo Giudice. “We need automation and work done in parallel from the beginning and not test as a stage later on.”