Industry Watch: The winding road to software failures

What began as a conversation about high-profile application failures and outages this summer took a winding road toward the root of what ails software development teams.

I was speaking with Mehdi Daoudi, who runs the experience monitoring solution provider Catchpoint, to try to better understand why website and web application failures continue on. He spoke about such things as the complexity of modern computing systems, which make those systems more prone to failure. He went on the say that the more complex systems are, the harder it is to document everything. On top of that is that the industry is now in constant release mode. “Everyone wants to push features as quickly as possible, but you cannot do that without things breaking, right?” he said. “Some of these eggs are going to be made into an omelet.”

We talked a bit more, and the discussion turned to the consumerization of software, and giving employees the same great experiences with their work tools as they expect from the applications they enjoy in their non-work lives. While organizations can control this when the applications are running in their own data centers, what can they do when they have to rely on cloud providers that they have no control over?

Daoudi said, “We’re starting to see in the workplace is a consumer mentality happen, where people at the beginning it was BYOD — bring your own device — but now it’s literally the freedom of using whichever tool you want to get your job done. CIOs don’t have as much control anymore over what productivity suite is used, so we’re seeing this shift. People used to call it shadow IT, but in my opinion this is ‘whatever it takes to get the job done IT.’ Even at Catchpoint, we have 200 employees, believe it or not we use 120 SaaS applications. When I saw the audit I was scared because now you have to take into account security and all that stuff. But it is what it is. You can’t go back to five applications; you can’t force people.”

With the proliferation of cloud-based productivity tools — Salesforce, Microsoft Office 365, Slack, Gmail and many others — when an AWS or Azure region goes down, your workforce is completely unproductive, and the costs of that are enormous. This, he said, is especially problematic in today’s world in which more and more people are working remotely, and using cloud services to get their jobs done. “But when you have that, you cannot tell people if you’re in headquarters you’re going to have super-fast access to X,Y and Z, but if you’re at home, or work at WeWork, or Starbucks, well, good luck, it’s on you. Everybody deserves a first-class experience, and that’s the mentality we’re seeing.”

From there, we talked about service level agreements, and writing them in such a way that they have real teeth and can hold cloud providers’ feet to the fire. The onus is on them to provide that great user experience, and if they cannot, they should be held accountable. In fact, Daoudi said one of his customers, Autodesk, was able to claim back millions of dollars from a vendor that failed to provide service at the agreed-to level.

Monitoring, Daoudi said, is built on four pillars: Reachability, availability, performance and reliability. “Companies try to do four things when it comes to monitoring in general. There is the reachability aspect: Can I get to Point B from Point A, and once I get there, is it up or is it down? Is it fast or is it slow, and then how reliable is it?”

Finally, Daoudi touched on what could be the root cause of software failures and systems outages. He put it plainly. People are tired. “People are literally exhausted. It’s this constant firefighting, putting out fires, at some point, you let your guard down. You become a little bit weak. In our SRE survey of 2019, one of the topics that came up is this fatigue, the stress. Some site reliability engineers even talk about PTSD to some degree. It’s just that people burn.”

So as systems grow in complexity, and organizations drive people harder and harder to release more features faster than ever — “Do more with less” is the mantra of the day — they risk burning out their development teams. That will put their systems and applications at risk, which in turn will put the organization at risk.