For those of us who work for a living and don’t have a television in our offices, we can still follow events like March Madness and the Olympics on the Web, thanks to the streaming video software created by Vertigo Software.
Vertigo, a consulting design and development firm based in Point Richmond, Calif., has been in the streaming video software implementation business since it first broadcast the 2008 Democratic National Convention live. Vertigo’s more recent contracts had its teams building the software that powered the online broadcasting of the Olympics, and for the second year in a row, the month-long college basketball tournament known as March Madness.
During the 16 days of the Winter Olympic Games in February, the NBC streaming video site created by Vertigo saw 19 million unique users, who watched hundreds of thousands of hours of video per day.
Bob Cowherd, senior software engineer at Vertigo, said that despite the fun of sports, webcasting these events were not trivial projects.
“With the Olympics,” said Cowherd, “there was very heavy involvement with the client, and the landscape was changing under our feet as we were moving. There were a lot of partners involved. There was another partner who did the HTML for the website. Akamai was the content distribution network. Conviva provided real-time health monitoring [for the systems].”
The secret to keeping all of those participants on the same page, said Cowherd, was opening the build process to all stakeholders. That meant all of the involved developers and clients could play with the latest builds and see exactly where the project was at any time.
Cowherd said that Cruise Control (CC) was a big part of that process. “We use Cruise Control. The CC server looks to see if it’s bad code being checked in,” he said. “If so, it doesn’t get published. There is an automated script to make sure things run smoothly.
“We also do strict versioning, so every time someone does a check-in, it gets versioned. Each one will have a build specific version number tacked onto it. This ensures you’re always grabbing the newest files. We know we’re guaranteed it’s the latest version. This saved friction and saved us from having to have a lot of back-and-forth.
“It’s not just a matter of focus on requirements, but also managing the relationships of each partner and all of the dependencies.” As this was a consumer-facing project, interface designers also had to have a large hand in the process.
Andrew Thompson, senior user-experience design manager at Vertigo, said that using Scrum helped to keep his team focused.
“In our case, the client tends to own the product backlog, which is a list of features the client is recommending. They really determined, sprint by sprint, what features we worked on next,” he said.
Thompson said the Olympics team, along with other teams at Vertigo, has also learned to keep the interface designers included in their projects right until the last minute. If a button needs to be changed at the eleventh hour, he said, having a designer still in the loop keeps processes flowing.
“Even though we may not need design staff on the project, we keep them on there. Things do arise,” said Thompson.
Make it real
Cowherd said that his mantra for the recent Olympics project was “make it real.” That sentiment comes from 37signals, a blog-renowned Web development company. Cowherd calls 37signals’ book, “Getting Real,” a touchstone.
“In the end, you can have loads of plans, loads of documents and big whiteboards, but really what you have to do is make it real as fast as you can,” said Cowherd. “In the end, the way you keep people on the same page is by making it real, then saying, ‘We’ve put code in place. Here it is. What’s wrong with it?’ Get it out there, make it real, make it available, and you’ll see what’s wrong with it as fast as possible.”
Vertigo uses Silverlight to build its streaming video applications, and it has been using the technology since it was launched. Cowherd and his team have since released an open-source streaming media framework for Silverlight, known as the Silverlight Media Framework.
Another key to success at Vertigo was that old mainstay: rigorous, proper testing. Cowherd said that this is particularly important due to the nature of sporting events.
“With a lot of these projects, some of the peak numbers of viewers are going to occur right at the beginning of the project. You don’t have a grace period,” he said.
“We do your basic lockdown period. We try to limit the churn, limit the changes. [You and the team] go from tell mode to ask mode. It’s the same as when you go into a test in school, and you know you aced it because you studied really hard. The tests you ace, you know you’ve aced them before you even get the grade back,” said Cowherd, calling out the importance of proper preparation.
Cowherd also claims that the team built “the most instrumented video player ever. I know, second-by-second, how many people are buffering. I can take a look at data, clip-by-clip, and see this clip has 10 times the error rate of others, and was probably poorly encoded.”
He also said graceful recovery from failure was essential in the software design.
“You’ve got to have a lot of points where you can recover from failure, both on the delivery end and on the player end too,” he said. “If a failure happens, what happens to the experience? If it does happen, it’s good for the player to try to recover by itself.
“We put up a nice-looking animation and say we’re going try to reconnect you. If you can’t reconnect people, fail in a nice way. Rather than an ugly error message, there might be something wrong with this one video only, so offer other options. We bring up a recommendations page, for example.”
And for those needing to avoid getting caught watching video on the job, Vertigo built a “Boss button” that allows the sportswatchers to hide what they’re doing by exchanging the video player for a static spreadsheet on a barren Windows landscape.
