For software developers at the San Francisco edition of QCon this week, the focus was on scaling software up and out. With talks on Big Data, HTML5 and developer productivity, the conference was bustling with real-world examples and explanations from folks who have learned from experience.
Folks like David Mortensen, engineering director at Facebook. During his talk on Thursday afternoon, he discussed many of the challenges that Facebook has encountered as it has grown from a small startup to a firm with more than 1,000 engineers.
To get to a thousand engineers, said Mortensen, Facebook set up an internal boot camp where it trained groups of incoming engineers to get them up to speed on the development tools, environments and practices they’d be dealing with. He said this directed effort for on-boarding engineers helped to ease the pain of rapid growth over the past seven years.
In the end, all of Facebook’s boot camp efforts were directed at improving developer efficiency. As Mortensen said, any discrete task that takes over two minutes for a developer might as well be a coffee break because it interrupts productivity in a significant way.
To this end, Facebook spent a lot of time and effort to streamline its development environments. Every developer at Facebook has his or her own development machine in the data center, which contains a complete running copy of the Facebook code. Developers can log in and manipulate the code, then glean the changes, and start pushing toward integrating them into Facebook proper.
The problem, said Mortensen, is that the development environments were initially extremely responsive because Facebook is written in PHP. That means changes would be instantly useable on a development machine without a need to compile. Unfortunately, as Facebook’s code became more complex, load times for development boxes became longer and longer.
To solve this, Facebook created HipHop, a PHP-to-C++ compiler. Once compiled, the C++ code would run much faster, but for the developers, every change now required a recompile, which could take up to 10 minutes. For Mortensen, 10-minute wait times were deadly to productivity.
After building an interpreter to stand in for the compiler in development machines, the wait time was cut down to around 10 seconds, but for Mortensen and the developers at Facebook, this was still too slow. The answer was to use a JIT compiler.