This year’s O’Reilly Velocity conference on DevOps and web performance began this morning at the New York Hilton. Keynote speakers briefed attendees on everything from “the serverless world” to data analytics, but a few talks focused on real-world scenarios that developers and IT leaders can learn and apply to their own organization in order to make sure DevOps is working in their favor.

Don’t gamble when it comes to reliability
Tom Croucher, site reliability engineer at Uber, was the first speaker at this year’s conference. He discussed a real-world example of how users can be impacted by a single small software issue, such as Uber’s outage in July, which affected more than 1 million riders and 450,000 drivers.

As a result of this outage, Croucher said Uber spent some time to understand what exactly went wrong with its systems to make sure it didn’t happen, since reliability is key. Like Uber, other companies might run into issues where their users will be negatively impacted, and so they should consider a few rules so that reliability is always in focus.

One rule Croucher discussed was making sure development teams know when something is broken. Companies can obsess over the data and fiddle with information, but this does not mean that teams know when something is broken. Organizations should know first how their users are impacted when something goes wrong, because knowing why something is broken is not the same as knowing what’s happening to your users, he said.

Another rule to follow is make sure mitigations are normal. For example, when moving traffic around as a result of a failed data center, development teams have to make sure they test. Testing is crucial, according to Croucher, so practice during peak times without letting anyone else know testing is being done. Also, make a plan and keep a log to understand how mitigations affect the system, load and capacity plan based on failover needs at peak.

DevOps, collaboration, and globally distributed teams
Ashish Kuthiala, senior director of marketing at HPE, hopped on stage next to talk about what he said to be the heart of DevOps: collaboration.

The big questions companies ask themselves, especially when they work at the global level like HPE, is how to get teams to work together consistently and with constant feedback loops.

One way that organizations can get their teams to work together and aim for that DevOps environment is to implement the use of ChatOps throughout all teams, which is what HPE is doing right now, said Kuthiala.

With ChatOps, HPE is able to collaborate with other teams throughout the IT department, but they also were able to augment its chat rooms with bots that were able to chat in real time. Bots are also set to retrieve statuses from other machines, so team members wouldn’t need to leave their desks to gather this information. Bots also send incident reports to other team members, who can then join a chat so they can all work on the issue together.

ChatOps helps HPE collaborate with other teams, and while teams are still siloed, it’s a step in the right direction for accelerating their DevOps transformation, said Kuthiala.

Two years in the U.S. Digital Service
Mikey Dickerson, who was heavily involved in rescuing HealthCare.gov, closed out this morning’s keynotes. He is a part of the United States Digital Service team, and he talked about his role in it and its role in creating technology experiences for the public.

The U.S. Digital Service is a network of startups across the Federal government, using design and technology as a way to deliver better technology experiences to Americans. The group has celebrated its second year, and the small team of seven or so has grown to almost 200 members.

A big question that Dickerson’s team had to ask themselves was what does it work on first? He said that the best way to figure out what projects to start with is to consider the greatest impact for the greatest number of people, and the likelihood of actually succeeding with the project.

One of the projects Dickerson talked about was one that Lisa Gelobter, chief digital service officer at the U.S. Department of Education, discussed briefly at last year’s Velocity conference. While Vets.gov is still a work in progress, much of the site has been revamped to give veterans a better user experience and make it easy for them to apply for healthcare.

Dickerson shared a few things the Digital Services team has learned that organizations can apply to their own projects. Point one is that an actual hard deadline and a sense of urgency are essential. He said that it is important to understand that huge institutions like government agencies were not built for innovation; they were built for durability and for a specific purpose. But, just because these agencies have lived for decades, it doesn’t mean they can’t change.

Other tips Dickerson shared were that successful projects defy organizational boundaries, and technology doesn’t ultimately change everything.

“We have tried many things, and this is the thing that does work,” said Dickerson. “This is the formula to repeat if you find yourself in a similar situation. Get a small [and powerful] team of people and make sure your people have access to the entire organization at all levels.”