The enterprise middleware landscape is shifting dramatically, with organizations pursuing a variety of paths — in both specialized managed services and DYI iPaaS packaged software – to get to the common destination of cloud-enabled middleware.

With cloud integration skills in high demand, new outside partners to work with, and even some LOB managers now creating business apps, corporate developers are feeling the impact.

Indeed, over the next three years, fully 76 percent of organizations plan to replace their existing middleware infrastructures, according to a recent study by the Aberdeen Group. While 43 percent will replace those systems only partially, 33 percent will do so completely, said Michael Caton, research analyst for IT at Aberdeen, in an interview with SD Times.

The study also found that 36 percent of organizations are running three or more middleware integration platforms. “As it turns out, in many situations, some of these systems came through company acquisitions,” noted Rob Consoli, chief revenue officer for Liaison Technology, the middleware managed services provider that commissioned the report.

Interviews by SD Times with developers, system integrators, and other industry observers affirm that enterprise middleware is migrating to the cloud fast.

Integrating external data
“Traditional middleware was the glue that held the enterprise together and let systems ‘talk’ to each other,” noted Marcus Turner, CTO and chief architect at Enola Labs, a custom corporate software development house.

Yet with ongoing demands to pay software license fees, aging systems can get expensive. Meanwhile, traditional middleware systems are geared to working with old-fashioned monolithic databases, at a time when organizations yearn to learn what they can so as to get ahead of the competition.

Furthermore, the programming skills needed to keep legacy systems humming are growing increasing rare, with developers trained in languages like COBOL retiring out of the workforce. “I’ve told some customers that they’re only a heartbeat away from needing to get new systems,” Turner remarked.

In contrast, while often also referred to as “middleware,” newer cloud-enabled infrastructures fulfill functions unimaginable 15 or 20 years ago. Essentially, these systems step beyond tying together on-premises applications to add integration with cloud-based information from outside the organization.

“Middleware platforms running in microservices environments are lighter weight, more decoupled, and more automatable. They also use standard HTTP,” pointed out Faisal Memon, technical product marketer at NGINX.

These newer middleware platforms are able to provide enterprise delivery of microservice data from both SQL databases and non-SQL data sources such as MongoDB. This big data is culled by highly distributed processing platforms like Apache Hadoop and Spark.

Hadoop/Spark helps out with microservices data in two major ways, according to Memon. “One of these [is related to] the sheer size of the data set. Using a single computer to do a simple math operation, like taking an average, would take months or years to complete. Using Hadoop or Spark with a cluster of computer resources would reduce that time to something more reasonable, hours or days,” he said.

The second way is that the microservices data set is not likely to be organized into a single structured database. “It’s likely to be scattered in different places and in different formats. Hadoop/Spark helps to bring the disparate data sets together.”

Middleware on the job
One of Liaison’s customers, a large pharmaceutical firm, recently built an app for accessing and analyzing microservices data from Twitter feeds using Alloy, Liaison’s cloud-enabled managed middleware platform. “The company wanted to know what consumers are saying online about its pharmaceutical products,” Consoli observed.

Organizations also use cloud-enabled middleware in synchronizing data between SaaS and enterprise apps. The endpoints to various SaaS applications tend to behave in different ways, meaning that without integration, LOB users are forced to separately access multiple sources of information. Meanwhile, more SaaS keeps springing up all the time.

Salesforce.com and other early SaaS pioneers have since been joined by hundreds of other SaaS, pointed out Vijay  Tella, a former Oracle exec and TIBCO co-founder who has since co-founded a start-up called Workato.

Experts also envision a surge in integration between enterprise systems and mobile and IoT data in coming years. Enterprise IoT data includes information from POS systems, for example.

To keep pace with newer distributed processing systems, cloud-enabled middleware  platforms  also use new  message brokers. Popular choices include RabbitMQ, Apache Kafka, ActionMQ, and ZeroMQ, for example. RabbitMQ, an open-source message broker distributed by Pivotal Software, uses the same type of message queuing architecture as traditional message brokers, but operates much faster.

Apache Kafka, on the other hand, stores messages in memory rather than queues. Like message queuing systems, Kafka is good at tasks like decoupling processes from data and buffering unprocessed messages. However, other uses of Kafka include log aggregation, operational monitoring, stream processing, and event sourcing, according to a use case scenario by the Apache Foundation.

If you’ve taken a look at one managed middleware service, you haven’t necessarily seen them all. With four years of R&D behind it, Alloy is “built from the ground up as a brand new development, leveraging a modern microservices and big data infrastructure designed to handle both computer application integration and data management requirements,” Consoli told SD Times.

Expertise in regulatory compliance is one key differentiator for Alloy, he noted. Liaison holds compliance certifications for HIPAA, European Privacy Shield, PCI DSS 3.1, SOC 2 Type 2, and CFR 21 Part 1.

Traditional middleware vendors like Oracle, IBM and TIBCO have also stepped into managed middleware services with add-ons to their existing product and services line-ups. So, too, has Red Hat, with managed services for its JBoss middleware.

The new Oracle Cloud at Customer, designed to be 100 percent compatible with Oracle Cloud, is for customers  needing on-premises data warehouses for regulatory reasons, pointed out Vikas Anand, VP, product management and strategy, Oracle Integration, Process and API Management Cloud Services.

Also among large IT vendors, Dell offers a managed middleware service called Boomi.

In an offshoot of the managed space, some application software providers now sell managed middleware services as an adjunct to SaaS.

Workato doesn’t sell its managed middleware services directly to customers, but to partners in the software industry, said Tella. The platform is in use by about 21,000 organizations.

Like TIBCO and Oracle, Workato provides corporate developers and even LOB managers with “recipes” they can use for quickly building workflow apps in human resources, sales, and other application areas. Workato uses Amazon’s AWS for data warehousing.

DYI iPaaS packaged solutions
Also aimed at easing integration, DYI iPaaS packaged solutions are provided as software rather than services. Top competitors in this arena include SnapLogic, MuleSoft, and Jitterbit.

Some contend that the DYI packages are less expensive than managed services, but Consoli counters that the cost savings are more than wiped out by organizations’ need to hire trained developers to implement the software.

Most of Liaison’s customers have experienced 50 percent or greater savings with Liaison’s managed services approach when compared to inhouse application integration, according to Consoli

One SnapLogic customer, a file sharing and content management company named Box, has deployed the packaged software in a predictive analysis implementation for use across multiple LOBs.

The system connects 23 different applications, includes 170 million data pipelines, and processes more than 15 million transactions per day, all with only 1.5 FTE developers, according to Alan Luong, senior enterprise systems program manager at Box.

SnapLogic’s platform also enables LOB managers to become what SnapLogic calls “citizen integrators” by letting them create apps without doing any coding. Adobe, another customer, now has about 500 of these end users in place.

Beyond costs, the complexity of needed integrations is another factor that enterprises look at when weighing whether to go with a DYI packaged solution or a cloud-enabled middleware managed service.

“Depending on the scope of the service provided, managed services can be a boon to organizations finding themselves short on expertise with middleware platforms of yesterday and even today. Managed service vendors provide numerous benefits to companies looking to save time and money on the various aspects of middleware management,” remarked Lucas Vogel, founder of Endpoint Systems, a systems integrator and developer of endpoint software.

“Building your own middleware or services layer gives companies full control over their service stack, but it comes at a cost with respect to having a dedicated DevOps and support team at the ready for managing the building, testing, and deployment of the middleware services, in addition to having a development team. Deploying to the cloud means deploying to a moving target, where everything from the tools to the platforms themselves undergo constant changes.”

However, to build smooth relationships with middleware managed service providers, enterprises should be upfront about the degree of integration complexity and the amount of maintenance expected of providers, he noted.

Opportunities for developers
In one of a series of recommendations at the conclusion of Aberdeen’s report, the analyst firm advised organizations to evaluate both iPaaS and  cloud-based managed services to determine whether they can free up time for IT to work on more strategic projects.

The report also uncovered internal skills gaps at many organizations for cloud-enabled integration. Enterprises voiced needs for more skills in SQL programming (61 percent); C#, C, C++, and Objective C (54 percent); API-based programming (53 percent); mobile deployment (54 percent); web development (47 percent); Java (40 percent); and data integration languages such as R (59 percent).

Additional needs emerged for more expertise in security and regulatory compliance, Caton said.