Erez Yarkoni is T-Mobile’s CIO. It’s a job that blends all the toughest problems of a telecom company, a consumer product company, and a full-time software development firm. In his own words, Yarkoni has spent his four-and-a-half years at T-Mobile “aligning and guiding technology solutions, platforms and architecture to a business strategy while leading the organization through a transformation spearheaded by a multi-hundred-million-dollar IT modernization program.
“The modernization strategy is reinforced by three activities: simplifying the hardware and software portfolio, upgrading and modernizing existing platforms, and introducing new platforms and enablers. The transformation has two dimensions: reduction of all IT cost elements, and time to market and enabling platforms that can support new business models including generating revenue from technology capabilities.”
We caught up with Yarkoni in January, and we asked him to explain how he manages to keep thousands of software developers across hundreds of platforms all moving in the same direction.
I know that T-Mobile really prides itself on taking care of its employees. Can you shed some light on how T-Mobile treats its developers?
I think, from a benefits perspective, T-Mobile is a great place to work. It’s a set of values that is driven throughout T-Mobile as a company and is part of T-Mobile as a culture. It’s a place where you have a voice within the scope of what you’re working on. It’s OK to voice your opinions, to have these conversations, to be empowered to drive change.
It’s not chaotic. It all works well within these concentric circles of scope. We don’t have a chop-shop mentality. I’ve worked in or worked with multiple carriers in North America, like Cingular before they became AT&T. I’ve worked with Sprint, TELUS and Rogers in Canada. This is a very intense place. We’ve done over 10 software releases a year, and about four major ones that touch almost every core system. We have 200-some systems, 40 that are core to the business. None of these is smaller than 20 to 30 projects, and typically touches 20 or 30 systems. These are fairly intense, and people do work long hours and weekends, but in an environment where they feel like they make a difference. They’re well educated about why they’re working and what they’re working on and why it’s important to our business.
Finally, development is a very centric component of what we do in IT. There is a lot that is circled around our ability to actually develop code. It’s important to our business, so it gives an edge.
#!
How is development organized?
We have two development organizations in IT, and then you can say there’re multiple development efforts happening in product development. In IT, we’ve divided it into any system that touches either a customer or an employee who touches a customer. So, retail systems (or) call center systems, we call them front-line employees. Also, any system that touches customers directly, like the website, the handset software, the portals. Then, we have the systems to support IT, like the billing systems. They’re all bundled into one group. The other development team handles our ERP and [business intelligence] environment; software that deals with our supply chain, the BI and the data warehouse.
Then we have the Q/A team. Each one of these teams employs about 220 to 260 front office, and about 650 back office, with about 400 to 350 augmented staff. They are flexible, but we have about 1,000 extra developers on staff at any given time. I would say our releases, when we work on more than one at a time (and we have a new release every three to four weeks), each one of these teams occupies 500 developers and another 180 to 300 or 400 Q/A guys.
Each one of these major releases—total hours translated to money—is probably multi-tens of millions of dollars per release, with, at times, over US$150 million of capitalized development.
Sounds like you have to be pretty agile.
We use agile in very specific places. One place we have been transitioning into agile is with overlaying all our UIs that touch our employees. There were multiple applications there, and the integration happened at a middleware layer. We are overlaying them with kind of a CRM layer. It’s more of a workflow engine, really. Call it a workflow UI. That environment is completely agile.
There are 12 agile teams in that system. They all take on projects within the team and in the system. They have corresponding businesspeople and dedicated Q/A teams to work with them on agile. Our billing system is complete waterfall; our ERP is mostly waterfall.
#!
I know T-Mobile has traditionally been a consumer-focused company, but smartphones have blurred the line between work devices and consumer devices. How has this changed your views at T-Mobile?
If you look at some organizations, like TELUS, they go specifically after enterprises, providing them complete enterprise IT solutions. We don’t do that. We just provide the voice product and the data connectivity.
That said, everybody has a smartphone, and I think inside small businesses a lot is done on personal smartphones. Let’s look at what Apple does in their stores or what REI does in their stores: You find enterprise mobility is driving productivity and unleashing other capabilities, like having no printed receipt. And e-mailing the receipt is much more than e-mailing the receipt: It opens up a complete universe of marketing. All of a sudden, you enrich your e-mail databases with addresses you couldn’t get before, and it’s all opt-in by the customer.
Has this mobile revolution extended inside your enterprise?
We have a very complex subscription product. Our enterprise mobility moves closer to the level of automating a medical infrastructure. It’s like subscribing to medical services or treating people. It’s form-intensive, requires credit checks and a selection of [software] packages plus hardware, and eligibility [information]. It’s not like picking an SKU off the shelf, scanning it and sending a receipt. The migration is a lot slower. Things we moved into mobile already are things like inventory scanning. Things we’re looking to move into mobile now are simple activation processes and taking payments in stores.
What’s at the heart of this mobile revolution for developers?
I think right now we’re looking at a true consolidation between Web and mobile. We’re getting to a point where we say there’s a way we can develop for the Web and mobile at once, and try to deal with the differences of the UIs or the users through rendering engines. I’m fairly careful about it, and rendering engines and proxies have been here for a long time, and I don’t think they’ve been very successful in the past. But now you can take a workflow that’s made for the Web and transition it into a mobile application fairly quickly.
Another thing starting to drive mobile is the evolution of development platforms. Mostly it’s been traditional IDEs, mostly in Java, and mostly unrelated to mobile. Can we transition our future developers to cloud development platforms, say in Node.js or Ruby or Python or Swing? Can we use Cloud Foundry to accelerate our development environment, or to build more cloud-oriented applications? What are these benefits of building code that is preconceived?
#!
What are you doing with big data?
We’ve adopted a couple of platforms into our BI environment. One is SAP’s HANA in-memory map/reduce. The other is Greenplum. We’re currently not really doing Hadoop itself. I am kind of waiting for Hadoop to be fronted and wrapped with more user- and developer-friendly frameworks. That’s kind of taking place. Obtaining and retaining a workforce that deals with Hadoop itself, I don’t know that an IT shop like T-Mobile is where those people will want to work.
The evolution is in the product providers like Greenplum or Netezza. They’re all working on systems that allow you to write your queries in a simple language, and underneath the engine will reduce it into Hadoop to execute it on a parallelized infrastructure.
One thing we’re very focused around BI is, as the ability to crunch a BI query becomes more real time, we’re more and more interested in integrating BI functionality into the transactional environment. That means BI happens inside a transaction, it reaches the transaction through the result of that BI.
Do you use open source?
The problem we have with open source—and we have a policy, from a risk-management perspective, from [parent company] Deutsche Telekom—we don’t run anything in production that’s open source unless we have a support agreement. You will find open source in a lot of our monitoring transactions. We have some Python in some middleware somewhere, but our core stuff we either write ourselves, or we buy platforms from IBM, HP, Amdocs or Oracle.
Have you standardized on an IDE or development environment?
We standardized on the configuration control. From an IDE perspective, the standardization comes within the teams. The organizational structure takes into consideration the natures of the technologies these teams use. Within those, there’s some standardization. Everyone who writes in Java writes in the same IDE. I think that we find a way to choke-in legacy. My intent is to find a way that in the future we do all our custom development in one PaaS platform. The standardization will come with it. There is never a 100% standard. If I get to 80%, I’m 100% standardized.
How do you make all these development projects work together? Dependencies must be terrifying to resolve with so many hands at work.
The things that I like to do in order to remove the friction associated with too many dependencies between the development organizations was to drive it toward a more loosely coupled architecture. A lot of the issues are not with the quality inside the applications, they’re with the contact between the applications. It’s the data flows and interfaces. We had to build better processes there, but that told me that layer of development has too many moving parts. We’re putting too much dependency of the outcome on the integration.
