Success comes with its own set of demands. Whether Ken Schwaber likes it or not, 2013 is the year of the “agile market,” and Scrum has reached the boardroom. The software development process Schwaber and Jeff Sutherland first presented at OOPSLA’95 is now the predominant technique used by agile software engineering teams. It faces new growing pains, however, as entire organizations try to adopt it, and new continuous delivery models bring complementary techniques (such as Kanban) to the fore.

Capitalizing on this momentum, Boulder, Colo.-based agile tool vendor Rally Software made an initial public offering in April 2013, and companies such as Atlassian and CollabNet could be next in line to IPO. Ever the “robe-and-sandals Agilista,” to use Microsoft principal program manager Aaron Bjork’s label, Schwaber shakes his head at all this.

(Scaling agile, according to Bjork: Getting agile to scale up)

“I was at a [Department of Defense] conference back in 2002. The CMMI people and a couple of us from the agile community were on a panel. At that time, CMMI was big, with lots of expense and consultants who would come in and make you Level 2, Level 3, etc. Bill Curtis and Mark Paul said, ‘We don’t think we have any difference in goals from people in the agile community. We all intend to improve the profession of software development,’ ” said Schwaber.

But CMMI had become commercialized, with an explosion of consultants and products. “The moment that happened, the initial purpose was lost. The guys on the panel asked us, ‘How will you cope with it when this happens to the agile community?’ ” Schwaber continued. That moment has arrived, bringing with it methodologists, consultants and vendors. But not without protest.

Scrum.org, Schwaber maintained, has “worked very hard not to come up with a methodology. I have a Scrum methodology that I developed in 2003. It’s very prescriptive: Do this and you’ll be agile. But I put it away. Someone said all these fads run about 10 years, it’s time for the next thing. But Scrum is based on values, like we stated in the Agile Manifesto. If the values take hold, we succeeded.”
#!From revolution to evolution
There’s no question the values have captured the imagination of innovators, entrepreneurs and financiers. Scrum calls teams to be iterative and incremental, to eliminate waste, listen to bottom-up intelligence, inspect and adapt, make people count, and use value-stream mapping.

“The Agile Manifesto applies to all industries. When we read it and its 12 principles, and switch each mention of ‘software’ with ‘customer-visible value,’ we have an elegant methodology that applies to all business,” said Joe Justice, whose day job is being a Scrum consultant for Jeff Sutherland’s Boston-based firm, Scrum Inc.

Justice spends nights and weekends running Wikispeed, an automotive-prototyping company whose volunteers aim to design and build the world’s first 100 miles-per-gallon commuter cars. The project dovetails nicely with his Scrum consulting, with customers such as John Deere coming into the Seattle Wikispeed workshop to build a car for two hours, then asking Justice to fly to India to help its engineers learn Scrum techniques.

According to Dean Leffingwell, creator of the Scaled Agile Framework (SAFe), Scrum now represents 70% to 80% of all agile practices. Though Schwaber has publicly criticized Leffingwell’s SAFe model as being a commercial methodology that bears too many similarities to the Rational Unified Process of the 1990s, he did concur that Scrum has won the hearts of most agile developers.

Ultimately the promise of Scrum is one of harnessing the power of change by building faster, but more importantly, it’s building the right thing. At a luncheon on a fall afternoon in San Jose, Rally Software CTO Zach Nies gave a brief history of innovation in different markets. His examples included Apple’s ever-shortening production cycle, Tesla’s attempt at revolutionizing electric cars, and even Webvan’s notorious breakdown based on a flawed assumption that shoppers wanted 30-minute deliveries. Each success story he shared, however, was a stark illustration of a Moore’s Law-like curve whereby the time from garage band to fame was shrinking ever faster, from decades to years to months.

Where old management techniques would “obscure uncertainty with details of a false precision,” said Nies, “What I’d encourage you to do is use tools and techniques to embrace that uncertainty.” Since no one’s business is getting less uncertain these days, he said, the trick is to find the sweet spot between discovery and complacency. “If you’re always getting the expected market result, there’s too much safety in your process. If you’re never getting it, you need more empathy with customers,” he said.

Rally has spent the last 12 years honing a set of Scrum-based tools and perfecting the techniques with which to teach them. And it’s no accident that the language the company uses to describe agile appeals as much to entrepreneurs as it does to software developers: In 2013, the company went public.

Betting on Scrum
The Standish Group estimated that agile techniques are being used for 29% of new software development projects, while IDC summed up the application life-cycle market (including configuration management, IT project and portfolio management, and automated software quality) at an estimated US$5.2 billion. According to Rally’s SEC registration statement, it claimed 154,982 paid users and more than 1,000 customers, including 36 of the Fortune 100 companies. With proliferating software and increased awareness of agile practices that improve efficiency, profitability and quality, that’s a promising market for the cloud-based tool vendor—one that has earned props from Microsoft and VersionOne.

“I won’t kid you that you can look at niche players that are making only an agile project-management play, and they are clearly ahead of us,” said Microsoft’s Bjork.

Andy Powell, VersionOne’s VP of customer experience, said the market sentiment is encouraging. “Rally is the first one to IPO,” he said. “We’re excited that they were able to go public because it shows how important agile is to the community as a whole. There’s interest for Atlassian to do the same from the investment community.”

Nor is Rally underestimating the threat from ALM suite vendors, according to its SEC filing. “Our primary competitors include private companies such as Atlassian, CollabNet and VersionOne, and public companies such as Hewlett-Packard, IBM and Microsoft, some of which can bundle competing products and services with other software offerings, or offer them at a lower price as part of a larger sale,” it read.

Part of the growth strategy for Rally, then, is to broaden its agile appeal beyond teams to the next level: Program and portfolio management. Of course, rising above the team level brings Rally and its ilk into contact with a new class of vendor, and more entrenched waterfall-based project-management tools. Being ISVs themselves, most of these haven’t been immune to Scrum’s call.

(Recap of Agile2013: Enterprise scaling and team collaboration)

In 2006, Bill Reagan’s team, Clarity PPM (a division of CA Technologies and a $4.4 billion IT management software and solutions company), began testing the agile waters. Clarity is a leading IT project and portfolio management solution with more than 1,000 customers, according to Gartner. Like many of its ilk, the product was built with waterfall, though, according to him, “We didn’t admit we were waterfall.”

However, executives were “used to getting visibility a certain way, with checkpoints, milestones and knowing scope upfront,” said Reagan, former director of product management for CA Technologies and now a Clarity consultant for Digital Celerity. But the dive into Scrum happened for two reasons.

“One, agile was an emerging market we believed in. And we also thought we ought to figure out how to get the two to work together; it was more like extending our capability, not replacing it,” said Reagan. As Scrum began to take hold, first in a pilot team in 2006, and then in a fully agile approach in 2010 to develop version 13 of Clarity, the company saw results.
#!“Beyond the final results, just within the team itself, agile was good for morale, energy and productivity,” said Reagan. “We got a lot more done. The reason for that is you’re very engaged. It’s a very collaborative, engaged development style. Everyone knew what everyone was doing.

“It’s also very adaptive… The team’s in control, so you deliver, as opposed to the big waterfall schedule, where no one is really responsible for the final product, just pieces of it—especially complex products such as Clarity. We’d bring customers into early reviews, and they like to be engaged, even if we get no value for it. But we did get early feedback and results, and we were able to deliver higher quality.”

The team was also able to build a SaaS piece on the Salesforce platform for Scrum teams using Clarity Agile. According to Reagan, the tool is for Celerity users who wish to connect the PPM to their agile teams, but it’s not applicable to the wider agile market.

Microsoft, too, has embraced agility, both for its tools and teams that build them. “Our strategy has always been, we’re an ALM suite. We want to provide you with everything you need,” said Bjork. “We also want to be open to other pieces. We have Visual Studio, which we think is the best IDE out there. We have three templates we ship with Team Foundation Server: Scrum, agile and CMMI. We introduced Scrum in 2010, and the response was phenomenal. It’s our most popular template—in fact our default template right now.”

A YouTube video from a few years ago shows a Microsoft team conducting a standup meeting in a hallway, chosen for convenience and to discourage lingering past the allotted time. According to Bjork, stand-ups don’t happen in the halls anymore, because there are none.

“Our new building on campus is built around team rooms. We don’t have offices anymore. We all sit in rooms that seat 12-18 people. You walk into any of these and see task boards and Kanban boards displayed,” he said.

Today, the TFS team operates on three-week iterations with a unique twist, said Bjork: At the end of the sprint, every team sends an e-mail to the broader group celebrating with a demo video. “This way, you get everybody contributing in a way that’s sharing more broadly outside of our team. It adds to a team element across the organization.”

Enter Kanban
With its massive user base, Microsoft has the luxury of analyzing developer behavior as it pertains to tools. “Team Foundation Service, the online version, over the course of the last two years has really come to life,” said Bjork. “We’ve been able to instrument it and study the data behind it at a deeper level than ever before with Team Foundation Server. We update it every three weeks, and we look at the data about who’s using what in the tool.”

So what aspects of Scrum are most popular, and which get ignored? “It depends. If you’re a new team to agile, most pick up right away with Scrum tooling; everybody uses the product backlog tool,” said Bjork. “A new trend we’re seeing is, as teams become more mature, they’re doing less task decomposition and moving to Kanban, where you don’t have iterations but have more continuous flow.”

(An in-depth look at the practice: Kanban: Is it in the cards?)

While task decomposition still happens for these teams, Bjork surmised it’s probably not to the granularity of pure Scrum. “We have data that proves it. Kanban is a more mature concept,” he said, describing a team’s ability to manage more advanced scenarios like a deployment pipeline.

“Scrum had more traction in the boxed product era; Kanban has more in the service or continuous delivery era. The channels to deliver software are so much more numerous than two or five years ago.”

Kanban doesn’t only work well in the continuous delivery era: It also fits better in the boardroom, where Scrum becomes awkward. “I haven’t seen companies actually doing agile at the portfolio level, but I’ve certainly been reading about it,” said Clarity’s Reagan. :It makes sense for the same reasons it makes sense for development: It’s about being adaptable, collaborative, iterative…”
#!Is Scrum’s glass ceiling SAFe’s open window?
As VersionOne, Rally, ThoughtWorks and other agile tool vendors try to bring Scrum concepts to the program-management level, they face serious challenges. How do they maintain the embrace of uncertainty and invigorating sense of empowerment while bringing ever higher layers of management into the fold? Not with Scrum of Scrums, a technique that brings representatives from each Scrum team together for a powwow.

“Scrum of Scrums is an interesting concept in that it works really well as a coordination tool where you look for impediments and coordinate teams,” said Kathryn Kuhn, agile evangelist at Hewlett-Packard. “Where it doesn’t really help is if you’re trying to gather status, and in larger programs, status is very important. Gathering up metrics and where people are and dependencies and compliance all come into play. Scrum of Scrums hasn’t always been helpful in harnessing all that, so we are starting to apply concepts from SAFe to manage our portfolio in a lean way without overplanning.”

Though it’s not the only approach vying for attention, SAFe is likely the most popular. Thanks to Leffingwell’s past lives at both Rational (where he helped develop the Rational Unified Process) and Rally (where he was chief methodologist), his approach is well positioned for success—but also an easy target for Schwaber, who said that following a prescribed process doesn’t require courage. Discovering what works best does.

(More tips for agile practitioners: Tech writers should be pigs, not chickens)

Schwaber’s disagreement does worry some, who also take him to task for comparing SAFe to RUP. “The architects of RUP are Ivar Jacobson and Philippe Kruchten, not Leffingwell. It does have a little bit of RUP smell, but I hope that 10 years on we’re beyond that. We have to help people understand SAFe is purely just a framework. We can’t coach everybody. If Ken [Schwaber] keeps telling everybody that this is RUP, there may be nothing that the Agile community can do,” said Ken Clyne, a transformation consultant for Rally.

VersionOne’s Powell pointed out that SAFe has roots in Scrum, just-in-time manufacturing and queuing theory: “Most principles underlying SAFe come from Donald Reinertsen’s book ‘The Principles of Product Development Flow.’ ” VersionOne too is integrating SAFe concepts into its program-level product, expecting to launch in October a feature called PlanningRoom Reporting to provide program and product managers with dashboards and visualizations of velocity, burn-down and other team metrics across a program of work.

According to Powell, a critical SAFe practice is shortening the project planning cycle to 10 weeks, spending two days planning on “potential shippable increments.” That has a potential to change company culture: “Once they realize they’re actually going to deliver, then they shift away from ‘I better get my big project in line’ to ‘What is the highest priority for the next 10 weeks?’ ” he said. And it’s intentionally not a 12-week, quarterly cadence, he pointed out. “It’s the sweet spot, where you can plan ahead enough but not have to replan in the middle.”

Powell expected rapid market focus and maturity over the next few years around discovery, program-level success and continuous delivery. “Organizations are getting better at figuring out how to blend user experience into their software development life cycle,” he said. “Another area where there’s a lot of room for growth, more success stories come out. Currently, a lot of failures coming out are in transforming at the program level.” And there are failures, if you know where to look.

What could go wrong?
Harvard Business School’s John Kotter is a renowned expert on leadership and change. According to Schwaber, “He and I collaborated back in 2011. He’s done [total quality management], lean, Six Sigma… Organizations bring him and his people in. His success rate was about 30% over five or six years with major organizational upheaval. Jeff and I thought about that, and our success rate was about the same. Getting an organization to change its culture is really difficult.”

Intuit, the same organization that Rally’s Nies praised for widespread transformation, Schwaber cited as a failure. “The effort to change at the enterprise level is so daunting,” said Schwaber. “We have helped hundreds of organizations become very agile, usually driven by someone visionary at a top level. When that person leaves, the excellence disappears. We had Intuit so they could do a release every day with a 900-person organization in San Diego. Then the general manager left, key VPs got promoted or left to go elsewhere. I’ve seen the same thing at Cisco and Siemens. I won’t even talk about Nokia.”

Schwaber’s latest book with Sutherland, “Software in 30 Days,” offers unflinching portraits of similar sagas, such as project management tool vendor Primavera (now owned by Oracle), where a successful Scrum transformation was abandoned after its champion, Bob Schatz, left.

Nies portrayed Intuit in a different light, however. “In late 2009, Intuit began using agile techniques on established incumbent products. One year later, they had $50 million in new product revenue. Today, they are running 1,800 concurrent experiments at any one time,” he said.

While contradictory, the moral of these two stories might simply be that agile takes an ongoing commitment.
#!What’s the motivation?
That commitment holds the hope of Scrum in an ever-changing technology landscape. The movement is essentially philosophical, with simple shifts in focus:

1. Shrink the change. When training companies in agile, Rally’s Clyne often touts the book “Switch: How to Change Things When Change Is Hard,” by Chip and Dan Heath.

“Sometimes you have to shrink the change,” he said. “The example in the book is about people who get in so much credit-card debt that they get themselves in trouble. Logic says to pay off the highest interest first, but sometimes it’s better to shrink the change and pay off the smallest debt first. You start seeing success and make that a habit.”

2. Let go of what’s done. A concept that often dogs developers, according to Microsoft’s Bjork, is completed work. In his book, “Professional Scrum with Team Foundation Server 2010,” he noted that tracking remaining effort on the sprint backlog is critical, even if teams “have an initial problem with letting go of completed work.” It’s important to do so, he wrote, because completed work estimates are rarely accurate, distract from the more important task of tackling remaining work items, and can pit team members against each other in a competition to complete the most.

3. Find the passion. In a world where everyone is potentially replaceable, does Scrum hold out any hope for meaningful work? Rally’s Nies thought so. “When I look at my past experiences, what contributed to burnout was building products nobody wanted,” he said. “Whereas if I’m passionate, I can work really hard. Also, I like to see the cadence side of it. A steady cadence is important. You don’t have these huge spikes of effort. I believe we’re all makers, and it’s a craft. If you take pride in your craft… nobody wants to build a crappy product.”

4. Make it fun. “Keeping engineers happy is about providing an excitement factor,” said Yoram Tal, senior director of engineering at Model N. “It’s engagement. To move from old technology to new language, new architecture, contributing to open source, potentially building something cool. Now we even have some engineers working on weekends.”

5. Use the surplus. But wait, there’s more, said Nies. “I ask teams a controversial question: As you get to 20%, 30%, even 200% increases in throughput, what are you going to do with the surplus? That’s a hugely important part of the cadence. Be selfish with it. Improve things for the team,” he said.

6. Improve the world. “The Scrum stuff, the whole agile framework makes companies go faster and cheaper and makes happier teams,” said Wikispeed’s Justice. “That’s fine, but what I actually care about is this idea: What if we put social good work in the backlog? If we’re losing sight of making the world a better place, then we’re just running the rat race with a faster rat.”

In a similar vein, Rally supports Engineers Without Borders, which aims to tackle global warming with technical aplomb.

The good news is, while noble goals may not always be Scrum’s purpose, they are often its dividend. Justice told a story about how his “boss,” Sutherland, described Scrum in purely economic terms. “I was really put off by that. It was so aggressively capitalist and divorced from meaning,” said Justice.

Later, Sutherland told him that Scrum has to be made attractive to the board and the CEO. “Good things will happen as an artifact of them chasing revenue. When you speed up a Scrum team, that’s where it gets all warm and fuzzy really fast,” said Justice. And that’s probably in the best interest of shareholders everywhere.

Scrum tools roundup
With Rally Software’s April 2013 IPO, the agile tool market has officially heated up. Here’s a thumbnail guide to its players:
• Rally is on a leading trajectory and enjoys the fruits of an agile internal culture.
• Atlassian may be the next to IPO, and has recently rebranded GreenHopper as JIRA Agile, which works with the JIRA issue-tracking and project-management tool.
• Microsoft Team Foundation Server and Service now come standard with a Scrum template, and they offer excellent scalability as well as integration into a full ALM suite, according to Aaron Bjork, Microsoft TFS Program Manager.
• VersionOne is expanding its focus on program-level agile with PlanningRoom Reporting and other Scaled Agile Framework reports.
• CollabNet is another company rumored to IPO or be acquired. Its ScrumWorks Pro continues to be a popular choice, with on-premise or cloud versions available, as well as Scrum training and certification.
• IBM Rational Team Concert now comes with agile process templates, including its answers to SAFe, DAD (Disciplined Agile Delivery) and Agility@Scale.
• Hansoft is popular as an agile tool among top game development companies. It emphasizes native speed for its customers with a new Mac OS X version.
• Axosoft has recently changed its pricing strategy, aiming for a broader customer base by lowering the subscription cost for its highly usable Scrum-based SaaS to just $7 per user per month.
• ThoughtWorks Mingle continues to delight agilists with its elegant, non-restrictive interface organized around cards and properties.
• Pivotal Tracker is a popular alternative for cloud-based Scrum, tracking everything from stories to epics, and also offering an attractive mobile interface for iOS7.
• Oracle has an Agile Product Management Lifecycle process available, but Primavera is perhaps at its most agile when integrated with Rally.
• CA Clarity PPM has been revamped with an accompanying Clarity Agile piece built on Force.com, but like other full-featured, traditional portfolio-management tools, it requires a major commitment for ongoing training and optimization.

Of course, there are plenty of simpler alternatives, such as spreadsheets, index cards and string. “Excel is still probably our biggest competition,” said Powell.