Now that we all live in this cloudy, serverless world, what in the heck do we do when something goes wrong? This was a major topic of discussion at QCon San Francisco’s second day. Today’s talks and gatherings focused heavily on engineering reliability at scale, and on how to build services that can fail gracefully.

To this end, Uber chief architect Matt Ranney took the stage a second time for today’s keynote. He discussed, again, the chaotic world of serverless hosting, and how Uber has built the tools it needs to manage the swirling mess of more than 700 services.

(Related: Uber’s Matt Ranney talks about serverless chaos)

Pinterest’s Yongsheng Wu, engineering manager of storage and caching, also spoke on the topic of building highly resilient systems. He said the photo-sharing board now boasts 100 million users, served via an amalgam of more than 100 services.

Those services are hosted across tens of thousands of instances inside Amazon Web Services, said Wu. “Our staff was mainly working in Python and Django on one monolithic application. We adopted Twitter’s Finagle, and later we adopted Go. When we needed more performance, we wrote several services in C++,” he said.

“It’s not about the language, it’s about using the best tool for the scenario or the use case.”

As a result of choosing tools and languages based on need rather than on enterprise convention, some of the systems inside Pinterest can handle more than a million queries per second, said Wu.

He said his team has architected for failure, and thus are able to deal with systems that break while in production. This is accomplished through a few systems and processes: dynamic service discovery, real-time configuration, caching, persistent storage, and asynchronous processing, he said.

Build your own PaaS
When it comes to constructing a cloud-based infrastructure that can reliably handle failure, Yelp’s Kyle Anderson told the crowds at QCon that, no matter what the project, their team is likely building a PaaS from scratch.

Anderson is on the operations team at Yelp, and he said that building a PaaS from scratch is not entirely wrongheaded; it’s simply something teams need to admit to themselves and embrace.

“Everyone builds a [custom] PaaS to some degree,” said Anderson. “There’s always going to be some process on top of your tools, so call that your platform.” He went on to state that, even if a team does choose an off-the-shelf PaaS, it will still require modification and custom code to work properly in their environment.

But writing your own PaaS allows you to customize it to your needs, said Anderson. The team at Yelp used this as inspiration to lay out their infrastructure’s fundamental tenets on paper.

Anderson said teams need to first survey what assets they have, with a focus on which aspects of the existing infrastructure are working. When Yelp made the transition to container-based infrastructure a year and a half ago, this is exactly what the team did. The pieces of its stack that worked and remained were Docker, Jenkins, Sensu and SmartStack.

Once the existing systems were tabulated, Anderson said the team wrote down some guiding principles for the PaaS they were building. Many of these didn’t become formally documented until the development had already begun, but these fundamental principles are highly useful for guiding new developers and making decisions, he said.

One of the key guiding principles for Yelp’s infrastructure was to favor declarative configuration over imperative commands. Instead of running a bunch of commands to deploy a service, for example, the team favored using the Git repository as the basis for pushing everything to production. That means even configuration files and metadata are checked into Git, and changes occur after those files are approved and pushed to production.

Anderson recommended Mesos for automated resource scheduling, and SmartStack for handling services discovery. Both of these were chosen because they are platform-agnostic, and especially because they are not tied into AWS. Despite using AWS heavily, Yelp fears lock-in there.

But the biggest form of vendor lock-in, said Anderson, is in service discovery. Using a system tied into a specific platform means your decisions later on will be limited, he said.

By using SmartStack, he said, his team can dynamically change and configure its infrastructure because every system in the network is automatically updated with new services listings whenever there is a change.

QCon continues through Wednesday.