Where does agile go from here?

Agile has been around for more than a decade now. It has proven itself at the team level and has scaled to the enterprise, but where does the methodology go from here in today’s modern software development world?

“The Agile Manifesto has stood up remarkably well for very broad adoption across a large number of organizations for 15 years now,” said Nathan Wilson, research director at Gartner. “It is always dangerous talking about changing the manifesto. It is like changing the Constitution or changing the Bible.”

According to Andrew Hunt, one of the 17 original authors of the Agile Manifesto and a founder of the Pragmatic Bookshelf, where agile falls short isn’t in the manifesto itself. The problems, he said, occur in the interpretation and implementation; but, that doesn’t mean he would not do anything differently if he could go back to 2001.

“If I had to do it again, I would add an emphasis on technical competence, and as a follow-up to that, on continuous learning for the individuals, team and organization,” he said. “Even though this is something we all understand and expect, I don’t think these ideas have been communicated well enough. Too many folks still think that an agile method is just a part you plug in and then you’re done. It’s not.”

Diego Lo Giudice, vice president and principal analyst at Forrester, adds that the principles and values of the manifesto are pretty complete and still very much valid for the main things that drive agile. “Everyone should read it first and understand it before thinking about how to apply agile in an enterprise or in development teams,” he said.

Improving the methodology
While the principles and manifesto may still remain relevant today, that doesn’t mean there isn’t any room for the methodology to improve and grow.

“In the general case, developers want to work in agile ways, but it is true that they are often blocked from doing so,” said Rob Purdie, agile coach at IBM’s agile academy. Those roadblocks include organizational structure and management buy-in.

“Agile could be made more accessible to developers if leaders would make the organizational and structural changes in support of agile ways of working,” he said. “It’s a serious investment.”

And since it is a serious investment, Roger Boone, a solution architect for custom software development shop Magenic, believes that there needs to be a bigger emphasis on funding and funding models. “There has to be some way for agile to address the big issue, which is budgets aren’t done sprint by sprint. You are not getting a new budget every two weeks,” he said.

“You have to have some idea of what a whole project is going to cost, and to be able to do that estimation upfront and get buy-in from the organization as a whole and as an investment; otherwise your project probably won’t get funded.”

Boone added that large organizations do their budgets at the beginning of the year, and they want to know how much each measured project is going to cost, which requires a lot of early planning that is more suited to a waterfall approach. He would like to see agile get better at estimating processes around how to identify things early and understand where the real cost is going to be.

According to Flint Brenton, CEO of CollabNet, upfront business planning is important in making agile more accessible and successful. He believes upfront planning needs to happen before developers actually start coding in order to help guide them to the overall project success, and to reduce the amount of rework and failures.

One area Boone thinks funding can be improved is for developers doing an initial sprint zero, also known as a backlog grooming workshop, where developers can understand the stories and get a better picture of their project.

Boone also notes that the manifesto is inflated, and in order to make it lighter, organizations have to adopt just the pieces that are the core values behind their organization’s process to make it work.

“I think the manifesto is bloated, but I don’t think anyone does all of agile in any one project,” he said. “They pick the pieces that really need and include the core components on the projects as they deem necessary.”

Another area that often gets overlooked is risk management, according to Boone. “Risk drops throughout the project, and your project eventually should be over when there is no risk,” he said. “Too many agile projects get stuck, and then they lose their way because something gets estimated with maybe 13 story points, that story gets stuck and it becomes a 40-50 story point. That is a problem because as they dig into it, it turns into a bigger issue.”

A focus on risk and an emphasis on having a Plan B in place should be integrated into agile to improve the process overall and really make agile move throughout the business, explained Boone.

Renato Quedas, vice president of solution strategy at Micro Focus, believes the question isn’t where does agile go, but where do agile practitioners go from here. “Agile practitioners need to evolve past ‘purism’ where they believe the whole company needs to follow agile practices in order to be agile, because the reality of such mentality is the very doom of agile adoption,” he said.

“Agile leaders have to realize what the end goal is enabling companies to achieve business agility by delivering better applications that meet business needs and customers’ expectations.”

Brenton believes the business side needs to increase the use of collaboration tools and community platforms to open the lines of communication and visibility between not only developers, but outside stakeholders such as business, marketing, support and IT operations as well. “Moving agile out of the narrow world of development will result in better organizational support that will lead to more success and deep adoption of the practice as a key business strategy,” he said.

Agile hasn’t been as inclusive as it should be, according to Hunt. “We need to involve everyone in the organization. That doesn’t mean a corporate-wide Scrum or standup meetings,” he said. “It does mean working together in a cooperative fashion to deal with the daily reality of a more technically sophisticated, hyper-connected world.”

With agile expanding beyond software development organizations, Gartner’s Wilson sees the need for more attention on the corporate setting and business value. He explains that agile isn’t just about delivering software faster; it is about making people in the company’s life better and bringing value to the business for the work that is done. “How can you incorporate this in a world where somebody has to approve the money? Somebody has to make sure that the project is doing what the business is guiding them toward,” he said.

Purdie suggests changing the language in the manifesto to be less software-development-specific, but he also thinks it is important to know the history of agile and where agile’s way of thinking and working came from.

The biggest thing for organizations to understand is that agile is not just for developers. Jonathan Atkins, director of design at IBM Watson, sees agile going through the same transformation that happened in the world of design and design thinking. Designers used to talk about user-centered design as an academic, technical approach to designing. Along the way, that idea of user-centered design transformed into design thinking once executives and industry leaders started to see the value.

“Design thinking is a brilliant rebranding and simplification of the basic principles of user-centered design,” said Atkins. “And design thinking made user-centered design comprehensible to anyone, and it packaged it in a way that it was about business results and driving business value. And that’s what I think the next step for agile is.

“Over the last decade, agile has been something that is complex, dogmatic, difficult to implement, requires major, major effort, and so it tended to stay just within development organizations. And it was thought of as, ‘Oh, it’s that thing developers came up with to make their lives easier but it doesn’t fit in the rest of our organization.’ I think the next step for agile is to go through that same transformation that user-centered design thinking did.”

Beware of unconscious incompetence
Eventually, agile won’t even be a thought; it will just become the way that we do things in the software development world, according to Hunt.

“Change [is] so constant that you no longer recognize it as change,” he said. “ For example, does anyone call it ‘e-commerce’ anymore? No. It’s just commerce. It’s just how it’s done. Does anyone say ‘I’ll go look that up on the Internet?’ No. I might Google for it, or more likely just ‘go look it up.’ The Net itself has started to become invisible, just part of the expected background. ‘Changing requirements?’ No. Just ‘requirements,’ and actually not even that. The idea of requirements as a static, definable entity is old-fashioned as well. So [agile] is no longer ‘a thing,’ it’s just how work is done.”

Gartner’s Wilson calls this a point of unconscious competence where developers don’t think about agile; it is just the way they work. But that can also be one of the biggest dangers of agile. “Agile is a continuous improvement process, so when you get to that point of unconscious competence, everything is just working and you are not thinking about what you are doing,” he said.

“You don’t have people talking about lean just-in-time manufacturing anymore because it’s just the way things are built, and one of the interesting conversations I have with various people that attend the Agile Conference every year is at what point does this just become the software development conference because everyone is doing it.”

That unconscious competence can actually become unconscious incompetence, where developers don’t know what they are bad at yet, Wilson explained. “The biggest danger to agile over time is that agile will just become something that people don’t remember that it is a continuous improvement process,” he said.

But Forrester’s Lo Giudice believes the future of agile is just more agile. “In the next two to four years there is going to be more agile than less. Today, about 20% of organizations have truly adopted agile and scaled it throughout the enterprise. There is still about 80% of organizations out there that are just playing with it,” he said.

“I don’t see any big change in terms of new paradigms. New things are going to come, they have always come around, and we always have had evolution. Evolution is going to continue, but I think a lot of the common sense that is coming with agile is not going to go away.”

Simple solution meets complex development environments
Agile focuses on faster time to market, shorter release cycles and better-quality software, but things can get complicated when businesses start to focus on more complex applications such as artificial intelligence and the Internet of Things (IoT).

There is tension when it comes to applying agile in an IoT environment, according to Gartner’s Wilson. “Connected devices have gone from the laptop you haul around to more and more the car you drive, the watch you wear on your wrist and someday maybe your glasses,” he said.

“It is all sorts of various end points, and a lot of them we don’t necessarily think as an Internet thing, so that is a big shift in the industry and a big shift because you develop things slightly different for the Internet of Things. There is a challenge for agile because one of the things agile leverages is that what we are building when we create software, it is free to build, meaning running the compiler is very cheap, so it is a different engineering process when the simplest thing is to build it and see if it works versus trying to build it right the first time like you would a bridge.”

Forrester’s Lo Giudice still sees room for agile in an IoT world. “There is still going to be integration issues, there still needs to be code to be written to kind of put these different intelligence pieces together so I think that is how product development will evolve, more development of agents and adding intelligence to these systems,” he said.

“This quick loop of agile from trying something, failing fast, trying it out, seeing what the feedback is and changing it, the basic principles of what we try to do with agile are going to stay around and just evolve.”

When it comes to artificial intelligence, Lo Giudice predicts AI will eventually become complementary to agile principles. “If you look at machine learning and deep learning systems that learn by themselves, that is going to be a different paradigm where what you are doing is you are training a network with lots of data and it just learns by itself on recognizing faces or understanding natural language,” he said.

“Agile principles will still apply because you can train it, you can give it data, you can look at your outcome, test it, see if it works, and if that doesn’t work you can just go back and train it with data.”