Software development has changed, moving from monolithic code blocks to a cobbling of open source and services. Delivery has changed, as organizations moved from on-premises servers to the cloud, and end points such as smartphones and all manner of IoT devices have become ubiquitous. How data is distributed and consumed has changed, as containers may need just a piece of data to operate, but that has to scale massively.
With all the wiring required to keep these systems up and running, while remaining highly performant, SD Times has recognized 2020 as the Year of Integration. It is the scope of what needs to be integrated, and the scope of the kinds of systems involved, that is becoming greater. Because of that, it’s getting harder to differentiate what is integration and what is just regular application development, or even data science and analytics.
As Matt Brasier, analyst in application architecture and platform team with Gartner for Technical Professionals, explained, “Nobody really writes systems anymore that just sit on their own and never talk to anything else. Everything has to be a part of this greater whole.”
In software development, the key technology for integration is the API, a long-used, well-understood way to bring data, functionality and services into your application. APIs and the services behind them change, however, so managing the APIs that you create internally and those you rely on externally, is important to ensure your application remains functional.
“That really comes down to a discipline we call full life cycle API management,” Brasier said. “The idea that the API, the interface in which you’re interacting, should be separate and have a separate life cycle from the back-end service implementation.”
Ani Pandi, director of solution engineering at integration platform provider MuleSoft, said, “That talks to best practices about how you do design of the APIs. For example, the UX/UI of an application is driven by consumer behavior. The fields that are identified in that particular UI is not exactly the behavior the back-end service is giving us. So we struggle with that design aspect, and then what happens is, we go back into this whole cycle of versioning, change management, where new versions of the API are created, and it’s not a very collaborative process. But if we start doing a design-first approach where the whole idea is to take design thinking practices to API life cycle, then you have the ability to look at the UX and the experience you want to deliver to your customer and take that and be able to create a model of your API that reflects that. From there, you figure out what do you want to implement from your API… is it orchestration, is it modernization of a back end, is it validation and enrichment and aggregation of certain information. Then you start really defining the API that is less susceptible to change.”
It is important to have consistency, and to do that requires a good versioning strategy for the API. And Pandi noted that the consistency standard should be accessible to everyone, whether it’s human resources or finance departments defining their own APIs. If they’re using a different versioning strategy, then you have inconsistency; you need to be able to define the strategy for everyone.
Further, Pandi pointed out, organizations need “a strong sense of how you do dependency management.” He explained that means when an API provider decides it has to version or create a new capability for the API, you have to be able to notify consumers of the API, both upstream and downstream. The comes from having an API life cycle management capability, and from that, effective communication and Agile development practices follow. “But today,” he said, “we don’t have sophisticaion in a lot of organizations to do that. The concept is to move away from a thinking of API management to a concept of API life cycle management, and that’s what we’re heading towards.”
But more than just joining services, APIs can be used by the business for competitive advantage. To do that successfully, though, requires a strategy.
“Just creating an API and putting it out there, there’s not much value in that,” said Pandi. “The value is, how do you take that to your partnership.” He gave the example of a bank that started partnering with retail property platforms, and embedded their API in their partner’s platform, so the consumer could not only review the homes they want to buy but also apply for a mortgage and quickly get approved for it. “That’s the experience where the bank has literally embedded itself in the customer,” Pandi said. “And that’s the kind of integration and being able to build the building blocks and unlocking data internally, and taking that experience outward through an API economy, is what we’re seeing as a differentiator and a capability that organizations are focusing on.”
One trend that is helping organizations deliver value to business through integration is the democratization of integration, according to Gartner’s Brasier. In today’s world, specialist teams of integrators are giving way to what Gartner is calling Integration Strategy Empowerment teams, which are creating best practices and platforms that enable non-specialists to create the integration flows they need. This is being enabled by platform providers offering integration tools with simpler user interfaces that don’t require weeks of training to understand.
Matthew Scullion, CEO of data transformation software provider Matillion, agreed that empowering “citizen data professionals” is where data integration is heading. “In the prior generation, the data warehouse was a specific team and a specific part of the IT department. Increasingly today, it’s the citizen data professional doing this stuff on behalf of the business. It kind of has to be. IT departments are turning into the service provider who are providing those citizen data professionals with the tools, and the citizen data professionals are actually doing the innovation with data.”
“The interesting byproduct of that,” he continued, “is they still have to load data, they still have to transform it, they still have to embellish it, because those things are computer science, and you still have to do that stuff, it doesn’t magically go away. You need to make the tools consumer-like.”
Alan Jacobson, chief data and analytics officer at Alteryx, agreed that this democratization is allowing people across different disciplines to do things “they’ve never really been able to do in the past.” He went on to say that the convergence of compute power becoming incredibly potent, data becoming much more available, people becoming much more data literate and technology becoming more accessible is having a dramatic effect on what’s available to people throughout an organization.
“There are a lot of different ways to think about data integration but the one that to a data scientist resonates the best is that the most challenging problems in the world — and usually the most valuable solutions — frequently come when that data scientist takes data from a myriad of systems, not from a single system,” Jacobson said. “When you really want to optimize the business, and you need some financial data mixed in with some customer data, maybe mixed in with some logistics data, and you need to blend all that data together to fully optimize the equation. That is a challenging data integration problem. And these systems historically were frequently built by very different areas of the business to solve very different problems, and they don’t naturally always key together. And figuring out how to prep that data, blend it together and get to insight can be challenging.”
As organizations move more of their workloads to the cloud, and look to use cloud-native solutions, there are opportunities to be had, but challenges to overcome.
Mattilion’s Scullion said the cloud offers companies the ability to compete using their data more quickly, at more scale and more cost-effectively than they ever have before. ”Companies don’t just want to compete using data anymore, they have to, as a competitive imperative. That drives a need to move apace. And if you want to move apace, you can’t rely on small numbers of individuals exercising high-end coding skills, because availability of skills and ability to innovate apace just aren’t there. You can’t get away from the raw computer science. Data is coming from different systems, it is in different shapes and sizes, it doesn’t necessarily have all the business rules built into it, and so you still have to do all that stuff we used to call ETL. And in our case, we call it ETL again. The need for cloud-native ETL is actually more present today than it ever was.” In the cloud, ETL must work alongside data warehouses and data lakes to maximize the ability to transform and use data for competitive advantage.
MuleSoft’s Pandi sees the need to build integrations that are flexible and don’t require teams to create new projects when things need to be changed. “I think we are at a juncture where organizations are starting to think that, we need to consider integrations to be a very strategic capability within the organization,” he said.
Gartner’s Brasier said the research firm is seeing a trend towards what it calls the hybrid integration platform. It’s a concept, he said, that explains that organizations will need more than one integration technology to solve all of their integration use cases. “You will have a mixture of specialist integrators and these ad-hoc integrators — developers and data scientists. Then you’re going to have a mixture of data integration and application integration and event integration,” he said. “You’re going to have all of these different use cases and there’s going to be no one tool that will solve all of these for you, so what you need to do is manage and give advice on a portfolio of tools, each of which is clearly defined for a specific use case.”
Mattilion’s Scullion said the point of data integration is delivering value. “The vast majority of business value being created in the cloud as regards data analytics isn’t migration and modernization projects. It’s net new questions being asked and answered in businesses in the cloud… questions a company wasn’t asking of itself five years ago but is now. Thinking about the citizen data professional, and these businesses not just wanting to but having to compete using data, faster than they ever have been able to before, fueled by the cloud, that’s most of what we see happening, and why data integration tools are really important.”
