Hybrid data connectivity is increasingly necessary in a cloud-based world where a considerable amount of data stays on legacy systems behind a firewall. But do you trust the cloud? Do your customers trust it? If not, how do you make secure hybrid data connectivity possible? Those are the challenges Progress aims to address with a new offering, Progress DataDirect Hybrid Data Pipeline. We spoke to Dion Picco, ‪general manager for data connectivity and integration at Progress in their North Carolina RTP office, about how they are refining their universal API approach to data access for enterprises and ISVs.

SD Times: Why is data proliferation a challenge? Isn’t it better to have more options?
Picco: If you look at the IT landscape, we’re in the middle of a disruption. With all the various types of apps and data structures available, the enterprise is in a weird state. The average company is running 15 to 20 Software-as-a-Service (SaaS) apps and also many on-premise apps. So data is stored all over the place and in many diverse formats. Thus, the internal analytics products are no longer self-contained databases.

We provide embedded data connectivity solutions for hundreds of ISVs around the world, and many of them are also facing this problem. Imagine you’re an analytics vendor, and you’ve built cloud-based or mobile applications. They’re no longer on-premise. But, most of the important data you want to analyze is either locked in another cloud like Salesforce, or hidden behind a firewall. The massive challenge of hybrid data is that to get that data, you’ve to either ask them to open a port in the firewall—a nasty conversation with any CIO, and one which delays your sale—or find another way to bridge the on-premise and cloud worlds. So our Hybrid Data Pipeline is a very powerful technology to capitalize on that opportunity.

It’s the same from the enterprise side. Enterprises still have a ton of legacy on-premise data. Tools that the line of business want to use are cloud-based as well. How do you let these apps communicate with data hidden behind proprietary APIs? And all of this must comply with industry standards, with high levels of security and fault tolerance. Progress DataDirect brings some level of harmony to this hybrid data challenge with our Hybrid Data Pipeline.

What about multi-tenancy? Is that an issue for ISVs?
Yes, when you’re dealing with a large ISV that operates their own cloud, they are going to be reluctant to have a dependency on another cloud service with another security model. The large cloud vendors such as Amazon, IBM or Microsoft can choose to deploy our hybrid data pipeline services and easily integrate the security models. It’s a big driver in how to build the architecture.

The Eurozone’s General Data Protection Regulation, passed in April, will take effect in early 2018. Are the European market’s privacy mandates a challenge?
Europe is a different ballgame. Data privacy and data residency, especially in Germany and France, are paramount. Where is data residing? What governments could put their hands on it? For example, as a company if you’re doing business in Germany, you want your data to reside in Germany.

Is the API economy helping or hurting?
Folks in IT know the that proliferation of cloud applications is becoming a challenge. It’s not a situation of not knowing the problem exists. It’s more about not knowing the pros and cons of a viable solution.

There’s a lot of messaging right now about APIs and the API economy. Now APIs are critical, but we are seeing 40 new APIs made public every week, and to task the IT department to deal with all these APIs—that’s way too much to do it by themselves. It’s tons of error handling, and each API is different. We can bring harmony to the API approach.

Can you give an example of the API proliferation problem?
This is common in business intelligence. One example is BOARD International, which is a BI and CRM vendor in the Gartner magic quadrant for BI. They have chosen DataDirect as a way to integrate these data sources. The customers they are trying to attract may have various flavors of Hadoop as they kick tires on a data lake. They may have custom app dev, Oracle, Marketo, Salesforce, Sugar CRM and other support systems.

To support this as an enterprise vendor, you need to customize code to hundreds of different sources. The cloud APIs alone, they’re the Wild West: Salesforce has a SOAP-based one with a query language called SOQL. Marketo has a SOAP API and a REST API (and no query language), so you need to figure out which one to use. Error-handling behavior is also very different among the APIs. Everyone is doing a ton of custom coding. APIs make a ton of stuff possible, but at scale they don’t make it easy.

When I think back to CORBA deployments in the late 90s, you’d go into a shop with either SQL Server, Oracle or IBM. When I visit a shop today, they have 20 SaaS applications, there’s SQL, Big Data, NoSQL—they’re servicing the developer more than they ever have. The landscape has been massively disrupted in the last decade; there are 500 different best-of-breeds depending on what you do.

How will your Hybrid Data Pipeline technology help?
The technology solves a few key issues. The first is to help you deal with the hybrid data world. The idea is to access data across the firewall without punching holes in the firewall. But most attractive for new prospects is solving the problem around APIs and the propriety interfaces façading around all of these. Progress DataDirect allows you to have a single, standards-based API—ODBC or JDBC, or OData if you want a more web-friendly way—that you can plug into every data tool. You can use these interfaces in a plug-and-play way with no coding.

For any data source—MongoDB, Eloqua, Salesforce—you now have one API. That’s really powerful. What you get is what looks like a single connector for a data access service. Architecturally, you have client-side plug-and-play with a very lightweight component for ODBC or JDBC—or clientless for OData.

As an ISV, you’ll probably locate your Hybrid Data Pipeline server inside of your cloud or on a commodity server inside the enterprise. Our on-premise agent can be deployed anywhere on your LAN (local area network) and be an end-to-end conduit for data.

Progress is more than just data, though. How does Hybrid Data Pipeline fit into the whole portfolio?
The way to think about Progress is that, at one point, we branded ourselves as an app dev company. But really, we’re helping companies transform their digital businesses. Say you have all your customer contract in PDFs somewhere, but you want to do self-service renewals. How do you authenticate and manage the transformation of those data and presentation layers? That is the concept that ties everything that Progress does together, from data access with DataDirect connectivity to our mobile platform, Telerik, and to our web content-management solution, Sitefinity.

So are these digital transformations merely aspirational, or do you have an example of one?
An interesting use case is what we’re doing with our Hybrid Data Pipeline technology to service Salesforce. Salesforce was a pioneering data application for customers already running SAP or Siebel. Salesforce had the foresight to build an external data strategy using OData, which is an open standard for queryable data.

Another great example is Intuit. They asked, “How can we simplify the sales environment so that we’re dealing with just one CRM?” They can’t just rip and replace Siebel CRM. The cost of ripping out Siebel would have been enormous, as there’s 20 to 30 years of built-up infrastructure there.

Instead, they made the data from Siebel available as an OData feed. That’s a very powerful story for them. If their sales representatives each wasted five minutes a day looking for customer data in Siebel times 10,000 reps? Now they are just going into Salesforce, and all the historic customer data is available in real time. There are no data storage costs. It’s truly a digital transformation, with DataDirect connectivity at the center of it.

At the end of the day, our goal as a company is to help our customers get where they want to go, and do so at a pace they’re comfortable with.