The digital transformation is something of a dream for the corporate world. Whether executives have grown jealous of fast-moving competitors’ products, or managers are sick of broken processes and hand-driven products, the end goal of just about every business in the world these days is to increase software development budgets, and to create more products that are driven by bits rather than wheels.

Take, for example, carmaker Volkswagen. For it the future holds terrifying prospects that can only be met with software-based solutions. One major aspect of this future is the active collection of user data.

When Tesla Motors began shipping its cars in 2008, it quickly became apparent that those vehicles were collecting large amounts of data. Tesla Motors founder Elon Musk famously got into a scrap with then British TV show “Top Gear,” citing the car’s data stores as a method of refuting the show’s relative dissatisfaction with the Tesla Roadster’s reliability.

Today, that data is not just being collected onboard, it’s actually being sent directly to Tesla, allowing the company to analyze its entire fleet’s behavior and shortcomings. While this has transformed the way Tesla handles its customer service, it’s also thrown other automakers into a situation where this type of data collection is now table stakes.

That’s fine to say on paper, but actually collecting millions of points of data every day from millions of users in millions of locations around the world is the sort of shoot-the-moon problem that would keep software developers and architects working briskly for a decade. For Volkswagen, the imperative is a great deal closer to the windshield.

Roy Sauer is the man at Volkswagen tasked with figuring out how to turn an automaker into a software maker. As IT CTO, he is responsible for the company’s internal systems and its customer-facing applications. Volkswagen has another CTO tasked with the technology of automobiles, but Sauer is the one forced to face a growing influx of data, and an expanding need for software.

Sauer said that Volkswagen created its first customer-facing application this past fall. The app helped Volkswagen owners work with their dealers to participate in the company’s massive recall of its diesel fleet in the United States.

While this was just a small app with a single purpose, Sauer’s teams are now pivoting to meet demands placed upon him by the business. One of those demands is to grow the company’s software revenue from zero to multiple billions of dollars sometime in the next decade.

That’s a major requirement that will require Sauer and his teams to be the drivers of the digital transformation inside of Volkswagen. Thus, working with Pivotal, Volkswagen has had to dramatically redesign its software organization. “As we are at a point where we change completely, we now want to move forward to a much more enhanced and increased software development approach,” he said. “Until now, we had much less capacity in software development.”

Given success on the app front, only half the battle is won: Sauer still has to deal with the influx of all that driver data over the next four years. Managing that data is a big part of his road map.

“This is one of the hardest topics,” he said. “What we know is that we will have a huge volume of data, and that is a big asset. It will be very important that we can handle this data. It starts again with our cars: We will produce as a group 10 million cars per year, and from 2020 on we will have 10 million connected cars per year, and we will have a huge number of devices in the Internet, like sensors for autonomous driving. They will produce a huge volume of data.

“The challenge is how to handle this data, how to use this data, and how to enhance further services for our customers. One critical challenging topic will be who actually owns the data and what will be legal. Where do we have to store this data? What is the data protection policy in U.S., in the EU and in China? This will be a really sophisticated kind of data management we have to do there.”

Dam transformation
One successful digital transformation took place inside BC Hydro, the power generation company in British Columbia, Canada. It decided to roll out IoT devices to its entire power grid as early as 2005, but the real effort to do so did not begin until 2011.

Pushing sensors into its network was no small task. While the software development behind those sensors took plenty of time, the project had help from Bit Stew Systems, a consulting firm that was recently acquired by GE in November. The efforts made on computer screens, however, were insignificant compared to the amount of work that was required in the field to deploy those sensors.

As British Columbia is quite large and includes some rather remote areas, one sensor, for example, require a full day’s drive into the wilderness coupled with a full day’s hike to it.

David DeYagher, manager at BC Hydro, said that the entire project required solutions that would last at least 20 years. “Our program was going to cost our company CA$900 million,” he said.

“With that we had to lay claim to $1.6 billion of benefits. The first part was the reduction in system losses using advanced equipment, and to put out intelligent apps to support system losses. This would do things like help us find where we had folks borrowing power from us and not paying.”

Just as with Volkswagen, however, once BC Hydro had deployed 2 million sensors to its grid, the real problems came around the management and analysis of the data.

“We went from 30,000 data points a day to over 270 million data points a day,” said DeYagher. “We needed someone with experience in terms of not only being able to ingest, but to also integrate some of the data into our source systems to drive the value. We also realized we had a plethora of systems our operators had to use. What the Bit Stew team offered was a single pane of glass—a manager of managers for tier-two and tier-three operators having single pane of glass to understand every metering device, every operating device that drove a lot of value for us.”

Which brings the digital transformation story back to the data, where many have said that the transition should begin. Jack Norris, CMO of MapR, said that bringing the data together can bring the organization as a whole together.

“The reason for convergence is that the digital transformation is about converging operations and analytics together: Combining analytics with operations so you’re not reporting; you’re adjusting the business as it happens,” he said.

“Less attention has been focused on the underlying data layer, but it’s the underlying data layer that enables real-time and missions critical capabilities. If you look at the dimensions of speed, scale and reliability, it’s very difficult to do all three across a large distributed cluster. I think regardless of what you call it,” said Norris, building a data lake can draw developers, analysts and other stake holders in to a single place to get their data.

“That’s just a small step on the journey, and by no means the stopping point because it’s about the ability to leverage that data in context and in real time. It’s the enabling layer for these types of apps and transformations. That’s where the second key of stream processing comes in,” said Norris, describing another enabler of the digital transformation.

“It’s typically these event-based data flows that are important to harness, whether it’s web events and machine sensors, or biometrics or mobile. If you can leverage those events and leverage those in context, and provide the historical aspects of data in motion as well as data at rest, that solves the problem of failure alert, monitoring and adjusting, supply chain optimization, ad optimization—the list goes on and on.”

That data is the heart of the transformation is echoed by Shaun Connolly, Chief Strategy Officer of Hortonworks. “The journey to digital transformation from our perspective is around the data and how do you assemble it in a way to fuel the digital transformation initiatives,” he said.

“For their digital transformation initiatives, enterprises are focused on cloud, IoT and Big Data all swirl around with the commonality being data. It isn’t necessarily about getting data in one place; it’s about getting data into places it needs to be.”

Another enabler for the digital transformation, said Connolly, is the merging of public and private clouds. Offering internal developers a quick and easy way to provision their systems is important, but being able to scale those systems into public clouds when needed is just as crucial, he said.

“That connected architecture on premise and in cloud…we’re increasingly seeing this as a digital transformation architecture,” he said. “You get some analytics, and in many cases want to act on real-time data and assemble it for cloud-based analytics and machine learning scenarios for a 360 view on root causes, in a manufacturing or repair type scenarios.

“We’re seeing it when you look at the videos of Hadoop Summit. We had well over 20 different customers sharing architectures, like Comcast, Ford or Capital One. When you look at their digital transformation initiatives, they’re increasingly inherently hybrid. It isn’t hybrid around ‘Let me burst my data to the cloud.’ It’s ‘How do I get my data sets where they need to be in a way that’s secure and well governed?’ Data science can explore, but you can also do that 360-degree view of customers on premise because you can mix it with data from the enterprise that may or may not make sense to put in the cloud.”

Customer transformation
Adam Seligman, executive vice president and general manager of Salesforce App Cloud, said that while digital transformation is a murky prospect, many companies are diving in headfirst.

“I’m actually blown away by how companies are leading into this customer-centric transformation,” he said. “It’s happening in every industry and every segment. Banks and healthcare service companies are all reinventing themselves in the age of the customer with the concept of digital transformation. I had a financial services institution said they used to wrapped their customers around their products, but now they want to wrap their products around their customers.”

The demand for the digital transformation of enterprises is not something that has been creeping up on companies for years, either, said Seligman. It’s something they suddenly realize is a business necessity: One day a competitor makes it a priority, or perhaps there’s a top-down mandate. However it happens, it seems to be more of a race than a saunter.

“They wake up one day and suddenly they’re in the software business. These are 150-year-old companies and they have to rethink their business with an app at the center of it. Their products and services are all threaded through this. It’s not just one app; it’s a whole cloud of apps. It’s connecting the customer with the community they participate in and making sure sales engages in a really customized way,” said Seligman.

“Fitbit is a brilliant device you wear on your wrist, but the true greatness of Fitbit is this community, the apps you use, and the communications from Fitbit keeping you motivated. That’s the brilliance that keeps you using the Fitbit,” he said.

And that’s the overall goal of the digital transformation: to turn great products into great communities that react to the customers’ needs in real time, without guessing and without humans driving every step in the process.

About Alex Handy

Alex Handy is the Senior Editor of Software Development Times.