When you’re watching the Oakland Athletics beat up on the Anaheim Angels on the MLB.tv streaming service, you’re probably not thinking about the software involved in that equation. But it’s the software that has taken 15 years of continual evolution to get to the point where, nowadays, baseball’s premium subscription service isn’t just a nice treat for the rich: It’s a genuine competitor for ESPN and Netflix.

Sean Curtis, senior vice president of engineering at Major League Baseball Advanced Media (MLBAM), is no stranger to running the MLB streaming service: He was there in 2000 when work began on building the product.

(Related: Continuous Delivery means getting code where it needs to go)

Curtis has been at a lot of other places since then, most notably at Amazon and Slingbox. But he returned to MLB a few years back, and today he heads up the teams that work on a great many of the MLB’s microservices and video products.

SD Times: The MLB has been on top of technological innovation for some time, now. It feels like they’re ahead of the curve compared to football and basketball.
Curtis: Digital events media has really been about figuring out how technology drives the best experience possible for our customers. We’re relentless about trying to understand the motivators behind our users.

We’re always looking for how emerging technology can help drive that, like the Ultimate Baseball Experience [sweepstakes]. That changes every day as the landscape changes every day. We try and be on the forefront. It is where we like to be.

Your streaming service is different from Netflix: It’s live and cannot broadcast games that are shown locally, plus it’s dealing with 30 different broadcast sources. How do you handle that?
MLBAM was formed by the 30 owners collectively coming together and saying, “Let’s centralize how baseball goes into the next phase.” That happened 15 years ago. Our subscription things were looked at skeptically when we started. Today it’s the over-the-top model.

People want to consume media wherever they are at any time. I would say the broadcast paradigm itself, and the advertising, and the home game rule, and how rights are distributed—we do have an increased awareness of things like location of home team affinity, and broadcast area. I think it is a little different than Netflix, where they have their library of video. There’s the emphasis on live programming, so it’s a little different of an animal.

Your services run 24×7, so do your admins and developers work 24×7?
We do have operations folks and traffic folks that are in 24×7, eyes on glass making sure production is happening. We use New Relic as a way to have eyes on those applications serving that media. The second anything happens with respect to distribution or access, New Relic lets us know that ahead of time. We catch that before our customers see an issue.

We have New Relic on all the various subsystems that go into the customer-facing notion of serving media. We’re making sure you’re in the right location, you are who say you are, and you have purchased entitlement to that media. If any subsystems experience some running anomaly, we find out fast and route to a different CDN or datacenter. Because we do have this fluid notion of live programming, as well as access to on-demand, we need a 24×7 incident response.

What other tools do your developers use?
In terms of IDEs, you’ll find people who still use vi, but a lot of our folks do use [the] IDE. For bug tracking and feature requests, we use JIRA and Confluence. In terms of coding and building, many of our teams practice Continuous Deployment. For the code, test, integrate, deploy and monitor life cycle, we have Jenkins doing tests. We’re using Git in most places. Developers code, create pull requests, and those changes are reviewed. They’re merged and Jenkins runs deploy and test.

We’re primarily a Java shop. There are some teams that have explored Scala. Those two, in terms of languages, are the things that save us time. What we’re continually doing is refactoring things that are more monolithic into microservices and dividing responsibilities. Across everything we do I’ll call it over 100 microservices.

Which of these tools is most important to your development process?
Git is pretty central to all our development teams here. It’s the primary method for iterating and modifying code, branching, creating pull requests. People almost never merge their own pull requests. That’s done by another developer, or a senior developer on the team.

Ultimately that merge gets checked in and Jenkins will pick that right up. We have a team in-house that helps with these common development paradigm tools. They stand up Jenkins for each team in the cloud that will take and run any unit tests that have been configured.

A lot of the load testing we do is scheduled with a team. That team has done some prototyping with Jenkins’ cloud test function, so load testing can be a function of the automated suite.

Then ultimately, the build team pushes to production. We use New Relic in the production ecosystem, and the monitoring feedback we get from those metrics really help identify bottlenecks.

The important thing is driving transparency throughout the entire process, and New Relic really helps us do that.

Did you learn anything in your time at Amazon that helped with the work at the MLB?
Amazon was an unbelievably great place to work, in my experience. The important thing they did right was to focus on the customer first. I’m trying to help our team understand we all win only when the customer has that compelling digital experience.

The other thing is, in some ways, I feel like I never left Amazon, as MLB has such a strong partnership with Amazon. I work with people there almost daily. It’s definitely been an integral part of our success.