We’ve all heard the old saying “A picture is worth a thousand words.” It’s trite, but it’s true. An image can cut through the crap and get everyone on the same page in an instant. In the software development world, examples carry the same power. That’s why I, like so many others, am a big fan of Specification by Example.

Using terms from object-oriented programming, I’m aware that examples don’t fully describe rules – they are an “instance” of a rule. However, like images, examples make business logic tangible. That’s why they’re so great at helping team members reach a mutual understanding. And – wait a minute – isn’t that exactly what functional test cases are? “Instances” of rules?

Specification by Example, 10 years later
I’ve been watching the Specification by Example community pretty closely these days, and I was excited to stumble upon Gojko Adzic’s great article “Specification by Example, 10 years later.” Gojko, who literally wrote the book on Specification by Example 10 years ago, ran an extensive research project predominantly focused on teams who use examples to capture acceptance criteria. There are tons of interesting findings there—I encourage you to give the complete write-up a close read.

I’d like to speak to a few points that really stood out to me.

 Confirmation that using examples boosts product quality
If you talk to teams using Specification by Example approaches, you’ll almost certainly hear anecdotal evidence that it has helped them improve product quality. Gojko’s study quantifies this with compelling statistics. He asked 500+ participants to estimate the quality of their releases based on the frequency and severity of production problems. He then segmented these results based on whether or not teams used examples as acceptance criteria. Those that were using acceptance criteria were nearly 3X more likely to report “great” quality than teams that did not.  

The Given-When-Then format reigns supreme
The study also found that “Given-When-Then” dominates example formats, by far. It’s used by 71% of the respondents. I believe this is because it’s the best balance of expressiveness and developer productivity. The study also noted that tables worked well for capturing large numbers of examples with similar structure. However, tables and Given-When-Then don’t have to be mutually exclusive. It’s important to note that tables have found their way into BDD Scenario Outlines, too.

 Democratizing test automation?
Like most of the innovative ideas regarding the SDLC, Specification by Example was first picked up by developers. Fast forward 10 years, and they still do most of the automation work (48%), followed by testers (31%). As a matter of fact, the “third amigo” — the business — is not really responsible for any automation… yet.

It will be interesting to see whether we get greater engagement from business analysts when no-code/low-code automation tools start integrating with BDD tools. The delivery team (developers + testers) and the business are already collaborating on the definition of acceptance criteria (the specification per se, and the basis for automation). In my opinion, this is a strong indicator that a good amount of business analysts (or at least business-focused testers) will contribute to the automation in the near future.

Specification without automation
Even if people don’t choose to automate the specified examples (approximately a third do not automate, according to this report), the practice of specifying the examples is still valuable. Even without automation, teams end up building comprehensive documentation of the expected behavior. Ultimately, they can make this documentation an integral part of their work/task tracking tools such as Jira and Azure DevOps.

What’s next for Specification by Example?
I’m extremely keen to see where all this will go. As you might have heard, Tricentis recently acquired SpecFlow, the leading BDD solution for .NET. The SpecFlow team is currently focused on bringing BDD and examples more natively into Azure DevOps, which is where all “three amigos” are managing their work. Regardless of whether the team decides to automate the specified examples, the practice of using BDD to implement Specification by Example here will invariably add value—leading to better software faster through increased collaboration and alignment.

I’m particularly excited about our plans regarding the scalability of BDD. As specifications grow, (object oriented) capabilities like modularization, re-use and referencing gain importance over the simplicity of structure. How can these concepts be established while still maintaining the lightweight, non-technical appearance of today’s Given-When-Then? Let’s find out together. Come join the SpecFlow community, and you can make a personal contribution as we shape the future of BDD together