For software developers at the San Francisco edition of QCon this week, the focus was on scaling software up and out. With talks on Big Data, HTML5 and developer productivity, the conference was bustling with real-world examples and explanations from folks who have learned from experience.

Folks like David Mortensen, engineering director at Facebook. During his talk on Thursday afternoon, he discussed many of the challenges that Facebook has encountered as it has grown from a small startup to a firm with more than 1,000 engineers.

To get to a thousand engineers, said Mortensen, Facebook set up an internal boot camp where it trained groups of incoming engineers to get them up to speed on the development tools, environments and practices they’d be dealing with. He said this directed effort for on-boarding engineers helped to ease the pain of rapid growth over the past seven years.

In the end, all of Facebook’s boot camp efforts were directed at improving developer efficiency. As Mortensen said, any discrete task that takes over two minutes for a developer might as well be a coffee break because it interrupts productivity in a significant way.

To this end, Facebook spent a lot of time and effort to streamline its development environments. Every developer at Facebook has his or her own development machine in the data center, which contains a complete running copy of the Facebook code. Developers can log in and manipulate the code, then glean the changes, and start pushing toward integrating them into Facebook proper.

The problem, said Mortensen, is that the development environments were initially extremely responsive because Facebook is written in PHP. That means changes would be instantly useable on a development machine without a need to compile. Unfortunately, as Facebook’s code became more complex, load times for development boxes became longer and longer.

To solve this, Facebook created HipHop, a PHP-to-C++ compiler. Once compiled, the C++ code would run much faster, but for the developers, every change now required a recompile, which could take up to 10 minutes. For Mortensen, 10-minute wait times were deadly to productivity.

After building an interpreter to stand in for the compiler in development machines, the wait time was cut down to around 10 seconds, but for Mortensen and the developers at Facebook, this was still too slow. The answer was to use a JIT compiler.

“We turned this on this spring, and got some awesome results,” he said. “When we went to the JIT compiler, there was cheering in the streets. But as a reminder, it’s never done. The site continues to get richer. This is one of those things that they continue to keep fighting on. It’s been a fun saga. With enough effort, almost anything that matters can be made efficient. You really want to make your engineers fast.”

Data structures
Another popular topic at QCon was the rise of new data stores. While many of the NoSQL and database companies were on hand, much of the discussion was around best practices for any data store.

Jeremy Edberg, manager of cloud operations and reliability engineering at Netflix, said he learned how to store data for quick access while he was working as an engineer at the news site Reddit.

Of the Reddit stack, he said, “There was Cassandra, Postgres and memcached. The biggest way Reddit does their resiliency is through the sharding of Postgres. Reddit splits writes across four master databases, which have different types of things. Each one has at least one replica in another zone. Votes on links only get one replica, while comments have 17 replicas spread across different zones. Reddit uses Postgres like a NoSQL store. There are just two tables: a thing table and a data table.”

Edberg said one of the keys to success was using consistent key hashing in memcached. “It’s important. If you’re using a naive hashing algorithm, which every memcached library uses by default, [you can run into performance issues],” he said. “If you add a server, it will shift three quarters of your data to a different server, so you invalidate three quarters of your cache. Consistent key hashing is the reverse: One quarter of your data will be in the wrong place. This makes it so you don’t put a huge load on your database when you add a new node. Protip: Use consistent key hashing. If you’re using memcached, you’re probably using a naïve one if you haven’t set it manually.”

Finally, Edberg said that designing for scalability from the start is key, but the initial number of servers is also an important decision. “Going from two to three is hard,” he said. “When you have one server, going from one to two is even harder to do, which is why I recommend going for three from the beginning. Scaling from three to six is a lot easier than from one to two or two to three.”

APIs everywhere
Another popular topic at QCon was the use of APIs. Mike Amundsen, principal API architect at Layer 7, said that enterprises are extremely concerned with governance and policy enforcement on APIs. But he also said that these new focuses on SLAs and productizing APIs has resulted in some interesting shifts in how APIs are handled inside big businesses.

“More often now, I’m seeing people in sales meetings who say, ‘I’m not the lead architect, I’m the product lead,’” he said. “They use that phrase. Even in the IT space, sometimes these product leads are inward-facing.”

In short, APIs are being tended to with the same care and feeding a regular software product would receive. This even extends to how companies approach API design.

“Other people showing up in the room are user-experience people,” said Amundsen. “I’m seeing people are using stories and personas for the API designs. User-centered design is a big thing for them for some of our customers.”

Unlike many of the API management themes out today, Layer 7 is not simply pushing REST, said Amundsen. In fact, one of the best practices his company suggests is to allow the API to be loosely coupled with its transport.

“You’ve got this compositional layer already that’s your business piece,” he said. “We’re telling people to actually consider the API as this thin veneer—like a UI—and a representation layer. Your components give you data, you compose that into something interesting, then you’re representing it to the client, either as a SOAP message or a JSON payload, or as REST.”