Solving problems with software. It’s what most programmers do all day long. But what happens when you come up against problems that never have been solved before? What if those problems were just leftovers from the last project? Or, what if those problems were already solved elsewhere, but in an entirely unrelated field?
For developers, reinventing the wheel is a risky venture: It takes time and effort to build any piece of software, but writing from scratch can provide some unique optimization and feature opportunities. It can also provide you with a chance to bring powerful tools to bear on an old problem. The residents of Caprica (on the hit TV show “Battlestar Galactica,” for example), had to find numerous ways to make their spaceships function under incredible stress and with very little resources.
Such was also the case of the massively multiplayer game of the same name. MMOs like World of Warcraft and Eve Online are enormous undertakings, requiring years of planning and development before launch, and even more human-years of patience and care once running. They’re similar to business services in that MMO servers must remain online all day, every day. Players tend to have day jobs too, so that means squeezing in game sessions during the wee hours and on weekends.
With that demanding a play schedule, and the need for multiple servers to be running simultaneously (each with around a thousand users signed on at all times), MMOs are big-business, highly available, almost real-time software. And that’s why the server-side of Battlestar Galactica Online is written in Erlang.
The logistics of Battlestar
Bigpoint Games is the publisher and developer of Battlestar Galactica Online, a free-to-play game where players pilot their star fighters through dogfights as either humans or their deadly cybernetic enemies, the Cylons.
Scott Guest is the studio technical director at Bigpoint, and he said the Battlestar MMO is “an Erlang product on the back end, in the server infrastructure. On the front end, it’s based on the Unity rendering client. It’s a thick back-end/thin-client design. Almost all of the work is done on the back end. It’s more of an input relay system on the back end and a visual presentation engine on the front end. The front end has less of the business logic of the game.”
So, why use Erlang, a language that’s almost entirely unused in the entertainment software development industry? “We had to make a product that has maximum uptime and scales really well,” said Guest. “We have a large flow of players into this game, and it never seems to stop. We chose Erlang because of what it was built for in the telecom space. The language was designed to solve the problem of uptime. There are paradigms in the language to recover from failure, and you have this observer pattern where you have processes that observe other processes, and if one goes down, it can be restarted. The word they like to use is ‘concurrent.’ ”
With thousands of Cylon raiders waiting to pounce on invading Caprica fighters, it would ruin the excitement of the game if one of those bad guys just sat there, motionless, his process hung in an endless loop on the back end. And that’s exactly what using Erlang and the observer model helps to prevent, said Guest. “Each entity in the game is its own group of processes, acting individually, but they communicate with each other through messaging. That’s one of the smart things about Erlang and this design. This is what Erlang allows us to do.”
#!
Guest said that if enterprises can learn nothing else from his operation, it’s that “Erlang is not just about making telecom software. It has far-reaching applications beyond switches in telecom, as demonstrated in this game. We’re cutting edge. A lot of people using Erlang now in the game space are using it to solve one aspect of their problem, like routing or managing load across servers. But we’ve gone one step further: We’ve built an application in Erlang. The lesson to be learned is that it can be used for other things, and it’s particularly relevant to concurrency.”
That’s not to say using Erlang hasn’t meant a few bumps along the way, said Guest. The biggest such bump has been finding Erlang developers. In fact, he said, it’s easier to train Erlang developers than to find them.
“Essentially, the product was seeded with people who had learned it, and then we reflected that with our staff over here (because it was developed in Europe),” he said. “It’s just easier to teach people, as long as you have enough time to teach them. Finding them tends to mean you have to go outside the country. There are a few pools of Erlang talent in the U.S., but those guys don’t realize their skills could be worth something outside of their current space.”
That’s a bit of a change for the games industry, said Guest, which has historically not been the place where concurrent programming theories were pushed to their limits. “Historically, the games industry was not as advanced as the enterprise industry,” he said.
“We didn’t have the compute power or the time and the budget. But now, some of the more challenging engineering problems are actually in games. There’s a relatively complex set of logic and behavior in an MMO. There are different types of trading and grouping that happen. This provides an interesting problem set that provides lots of creativity and complex solutions. It’s not just the scale problem. I think, historically, enterprise was largely about scale problems. Now, in videogames, it’s scale problems plus game-related engineering problems. It’s about how to deal with large amounts of transactions, of data, of interactivity between players in real time.”
Unity of command
Agility is also a key to the success of Bigpoint’s projects, said Guest. And while most enterprises are already familiar with agile, they may not be moving at as fast a pace as Bigpoint.
“We’re very agile, especially in a live product,” he said. “We have to respond to what the consumer wants. Agility is the key to that. We have a two-week release cycle. We listen to analytics-based feedback, and we shuffle that into a priority of development on a two-week cycle.”
That agility is also helped by agile tools. On the front end, for Battlestar Galactica Online, that tool is Unity, the popular 3D game development IDE from the company of the same name.
Unity is built to be agile from the ground up. Unity’s CEO David Helgason said that Unity is a first-of-its kind tool for the game industry, and for the development world in general.
“You could argue that in the game industry, Unity is the first to release a simplified tool at this level, where it’s not just for the game industry. An individual developer at a company can get [a project] done in hours or days rather than building with a team from the ground up,” said Helgason.
He said his game tool company is modeled after the ideal, rather than the reality, of Google. “We didn’t model our culture on Google, but we modeled ourselves on a platonic ideal of how Google might be. We’re giving engineers the power to come up with their own solutions toward a big goal, rather than toward a detailed road map. That’s something anyone writing software could consider, but I understand it might not be replicable everywhere,” he said.
The degree of control given to developers and engineers at Unity is essential, he added, because the team is only around 160 people strong, and the demands of the Unity codebase are growing every day. Today, Unity developers can build their game on a desktop Mac, and deploy the binary end product to Android, iPhone and any Web browser that supports the Unity plug-in. And the Unity team is planning to make its binaries deployable as Flash applications soon.
“It’s not just engineering,” said Helgason. “We’re building software for a very complex market. We do it quickly. We have to work with chip companies, and then we sell our solution into around 20 different industries and sub-industries.”
With so many branches of code to keep track of, that individual ownership of problem-solving helps to keep the pace of innovation and agility fast.
“Basically, we’re doing all this complex stuff and we don’t have enough brain power,” said Helgason. “There is no one who knows all the stuff that needs to happen. The solution is to let the decisions be made where they’re executed, and maybe partly communicated upwards. That means there’s stuff that happens at the company that no one knows about, which is good because people are passionate about it. We don’t use agile in a formal way; we don’t enforce that. A lot of organizations go to agile from a more formal methodology. You find freedom in agile or Scrum, and when we talk about Scrum, Scrum is almost too enterprise-y to some of our people. We’re coming from such a ridiculous degree of freedom already.”
Unity has certainly made good on its promises, likely thanks to that freedom in the development cycle. In 2010, Unity won a Technology Innovation Award for software from the Wall Street Journal, and the company has begun shifting its monetary focus away from the purchase price of the IDE and toward its online store, where developers can purchase prewritten code. The store offers preformed components, such as better 3D texturing procedures (known as normal mapping), and even pre-designed 3D objects such as cars, guns and monsters. Unity gets a cut of all sales in its store, but it’s the end developers who are filling that store with products. Some of those third-party developers are already making considerable profits from the sale of their virtual assets.
But what Helgason said enterprises can most learn from Unity as a company is its focus on the big, overall goal. “We don’t do a lot of Nerf guns and partying stuff…What we actually do, and what we get right is: There’s a high-level goal. I used to be in enterprise software. It’s become so milestone-focused. If you’re only looking into the next few weeks’ features you have to do, sometimes you forget to put up the really high-level goal. I think enterprise managers—just because it’s not in the culture—they didn’t put up the big goal. In Unity, we want to democratize game development. That means someone in the organization who comes across a decision, they can answer it by themselves by comparing it to the mission. Having a vision-driven organization is really healthy.”
