Platform engineering has become increasingly important for businesses as platforms have become more complex, spanning DevOps tools, APIs, and other components necessary for effective software development. It’s a delicate balancing act as developers have been calling for more simplified navigation throughout an organization’s platform.
According to a whitepaper by Humanitec, just five years ago, platform engineering was not a thing people talked about.
In the last decade, the concept of DevOps was all people thought about, ever since Werner Vogels remarked “you build it, you run it” at an AWS launch in 2006.
This shift to DevOps caused a notable shift left in roles, where developers are now responsible for more aspects of an application’s life cycle and delivery workflow, all while the industry moved to more complex microservice architectures and technologies like Kubernetes, GitOps, and Infrastructure as Code (IaC), the report added.
“Platform engineering emerged in response to the increasing complexity of modern software architectures. Today, non-expert end users are often asked to operate an assembly of complicated arcane services,” said Paul Delory, vice president analyst at Gartner. “To help end users, and reduce friction for the valuable work they do, forward-thinking companies have begun to build operating platforms that sit between the end user and the backing services on which they rely.”
The more successful engineering organizations invested in building Internal Developer Platforms (IDP) resulting in better performance on all DORA metrics. The IDP is the sum of all tech and tools that a platform engineering team binds together to pave a golden path or paths, according to Humanitec.
Gartner forecasts that by 2026, the majority of software engineering firms (80%) will have created platform teams to supply mutually accessible services, components, and tools for application delivery. This will ultimately address the key challenge of collaboration between software developers and operators.
Lack of structural discipline is causing many platform problems today
The default assumptions and behaviors around what came before platforms were basically very bureaucratic or consultative, but not in a good way, according to Charles Betz, vice president and research director at Forrester.
“These groups have always been saying, well, if you need a computer, a virtual machine, or some compute resources or a database, you really need to have a server engineer and database administrator there to make sure you don’t get yourself into trouble. We’re going to be intimately involved in your technical design,” Betz explained. “We have overloaded people and it’s going to take long and unpredictable amounts of time for us to give you the designs and the analysis you need. Then oh, by the way, no, you can’t have access to any of the resources you need until we approve. And this is what leads developers to really dislike classic, traditional IT.”
However, Betz said that he also sympathizes with the IT side of things because IT professionals have typically been overloaded. They are often subject to unreasonable demands, and there have been histories of people developing systems with horrendous architectures, and then insisting that the infrastructure group take production ownership.
“The reality is that the long periods of delay that the classic infrastructure group imposed simply is unacceptable in the modern Agile world,” Betz said.
He added that people love to point to companies such as Netflix to try to mimic their way of being Agile, but those are the companies that are highly disciplined from a platform engineering perspective.
“Netflix did not have autonomous product teams at a low granular level choosing willy nilly what products to use. That’s one of the myths that we often have to deconstruct when we come to a client. People say, well, all the product teams insist that they’re autonomous, and they can do whatever they want. And they point to Agile and various things. I’m like, you know, that’s actually not how it happened,” Betz said.
The companies that succeeded imposed architectural discipline although they might have not called it that. But the bottom line is that they didn’t tolerate a lot of unmanaged variability and sprawl and that’s how they managed to be successful, according to Betz.
Humanitec’s DevOps Benchmarking Study 2023 found that giving DevOps tasks to developers as a way of implementing self-service is often executed poorly in many organizations. An IDP can enable developers to have greater self-service capabilities with the ability to spin up environments, deploy, roll back, and make changes to the architecture without relying on Ops.
It is important, however, to keep the relationship between Ops and developers close while maintaining a separation of concerns. Through this, both teams can work together while having distinct roles.
The Humanitec report indicated that successful teams manage their app configurations across the entire organization in a standardized way. They also handle app configurations and infrastructure dependencies in the same manner, knowing how to distinguish between environment-specific and environment-agnostic configurations. These teams are more efficient at creating new environments and are able to provide more self-service when it comes to deployments, provisioning infrastructure, and assigning the infrastructure.
The path forward for platform engineering isn’t about changing the engineering
As a whole, platform engineering is now undergoing a lot of experimentation and there isn’t an agreed-upon best practice out there yet. However, the biggest challenges facing platform engineering don’t always deal with the “engineering” part at all, according to Forrester’s Betz.
“When I went to the DevOps Enterprise Summit last fall, I talked to as many people as I could who were having success in platform engineering, and the one thing they all had in common was they were figuring out some way to bring product management principles to platform engineering,” Betz said.
Platforms are products, according to the authors of Team Topologies, which suggests that the principles of organization, self-correction, monitoring environment, and setting standards are applied to the team.
The challenge is that platform teams are now being created from the existing, outdated IT and operations organizations. These organizations are best suited for engineering purposes but are not necessarily well-equipped for the other tasks of a platform team.
The problems are not in feasibility, which still continues to require high levels of engineering excellence, but rather in adding value, viability, and usability, according to Betz’s article, “Platform Product Management Versus Platform Engineering.”
This platform model encourages seeking automation wherever possible and managing queues anywhere else, making the internal offerings have advantages in terms of access to capital, reduced transactional friction, and maintaining high-quality service. Organizations should also implement explicit service design thinking in which employee net promoter score (eNPS) is tracked and customer journeys are understood, according to the article.
“To create a platform that is useful, you need to understand what is it the actual people you’re creating this platform for want and make sure that you’re regularly talking to them, iterating with them like you would have with any product backlog, product development, and product in the world of application development,” said Daniel Betts, senior director research analyst at Gartner. “You’re treating this in a similar way, you’re having to create a platform as an agile product.”
Most of the teams that the products are targeted at are software developers, application developers, infrastructure engineers, and someone who’s creating code, creating assets for the business.
“These people want to be able to create software or applications to deploy to a platform. They don’t want to have to think about tools, technology, governance, change management, all of those things. They want them all to be sort of made available to them and they want to focus on writing code,” Betts explained.
Often, platform teams are composed of a software engineer or two, as they add expertise to the benefits of writing machine-controlled code, code reviews, automated processes, and reusable components. They also aid in teaching scripting and coding best practices. Additionally, SREs can be part of the platform teams, as it is key for a successful product, Gartner’s Betts added.
On top of that, organizations are having to develop platform engineering talent internally because there is a skills gap and they can’t find someone in the market that will know exactly how their platform works, according to Forrester’s Betz.
“You have to do things like take somebody who’s a good technical engineer, and you’ve got to pivot them a little bit into somewhat less technical concerns like developer experience, product management, and products need to be valuable, viable, usable, and feasible,” Betz concluded. “We need to see more platform product managers.”
