SPOILER WARNING: This article contains major spoilers for the TV show “The Good Place.”
In the television series “The Good Place,” four people die and are told they are in heaven — The Good Place. But their time is marked by a series of escalating annoyances that finally makes one of them realize they are not in fact in heaven, but are in a living hell — The Bad Place.
Many conversations I’ve had over the years about the benefits of the cloud, microservices, shifting everything left into development, and giving developers the keys to operations, have made us as an industry think we are getting into The Good Place of faster application delivery, better value and experiences for end users while silos between developers, QA, security, IT and the business are broken down.
But to get to this technology heaven, developers are being asked to do things they aren’t trained to do regarding testing, security and governance. And organizations have been slow to invest in the training necessary for developers to be successful. This clearly creates risk for those organizations.
Because much of today’s application development relies on open-source packages and features accessed via APIs, developers are using a lot of code they didn’t write, and have to have faith that those open-source packages are updated as vulnerabilities are found.
Meanwhile, traditional IT operations has been sidelined, so the precautions they would take before releasing an application — making sure the deployment doesn’t break anything, or introduce vulnerabilities, or that the application will perform well and give a great user experience — are being dealt with after the fact.
And, as some risk assessments determine that a vulnerability in an app needn’t be fixed because while it’s an opening, it doesn’t have a path to any critical data, mountains of technical debt are inexorably growing, leading to potential damaging data breaches when changes to software packages now make that vulnerability a path to the company jewels.
In this new development world, where teams are given the autonomy to decide what their next projects will be and how long it will take them to finish, managing all this has grown increasingly difficult. This has not broken down silos, but has served to create more.
Every part of the organization is in each other’s business, using different languages and terminology, serving different ends, while fighting for their priorities rather than trying to achieve the same goal.
Like it or not, these silos remain. Scott Moore, director of customer engineering at Tricentis, told me: “If silos didn’t work at some level, they wouldn’t exist.”
Given all this, I’d say you can make the case that DevOps is, indeed, The Bad Place.
But, just as the characters in the television ultimately learned to work together as a team to finally reach The Good Place, I’m supremely confident our industry will find its way there as well.
But the pain will continue as we work to figure it out.
First and foremost, organizations need to make serious investments in training. This is a new world, and you can’t shift tasks left onto developers who historically have never had to deal with things like security, testing in the pipeline, containers and Kubernetes, and writing infrastructure code.
Part of the frustration developers feel, as reflected in the numerous surveys we see on the subject, is a lack of support from the C-suite. Developers do not believe their companies are dedicating enough resources to training, leaving them on a limb when something goes wrong.
Investing in training is most important.
Tricentis’ Moore said companies he speaks with about DevOps ask who’s doing it well, and how they do it. When he asks them where they’re at with DevOps, they either say they have it under control, or “they say we’re on a journey, which means they really have no clue.”
SREs, he suggested, are seen as the “find it, fix it masters of all,” when in fact they’re high-paid engineers who provide tech support in high-priced war rooms. He suggested creating the title of software life cycle engineer to handle the specialties such as test, security and performance. These engineers, he said, “will know how to improve performance before they write a line of code. Same with security.”
Further, organizations need to adopt progressive delivery to further protect themselves from delivering flawed or even dangerous software to their users. Using feature flags and releasing to a small cohort of users least likely to abandon you over a bad delivery — and having the ability to quickly roll it back if it does fail on any level — can protect you and your customers as well.
At the end of the day, lifting the burdens off untrained developers and putting them onto life cycle specialists may not solve everyone’s DevOps problems, but it will get them closer to The Good Place.
