Fellow architects and designers, I fear that we as an industry are moving our applications and data into the cloud without first having mastered service-oriented architecture, the basic discipline of building distributed systems. In the process, we’re setting ourselves up for failure.

If your experiences have been anything like mine, SOA has been a disappointment. Think back to a decade ago. We were promised lower operating costs, improved business agility and reduced operational risk if we would only adopt SOA. Filled with enthusiasm, and with the encouragement of reputed vendors and analysts, we persuaded our businesses to invest generously in technology and training around service-oriented architecture.

We’re ESB-ed and Registry-ed to the gills. But now, 10 years later, when we total up the benefits we received by adopting SOA—and there have been some—they haven’t been stellar. New systems still cost a lot to build, existing ones still cost a lot to support, it still takes time to create new business capabilities, and our systems still experience unexplained outages. So the promise of SOA hasn’t been realized, if we’re thoroughly honest with ourselves. Where did we fail?

SOA itself is not at fault. The basic principle behind SOA is “loose coupling,” which is based on perfectly sound software engineering experience. Think of “loose coupling” as the absolute minimum required set of dependencies that should exist between any two systems. Too many dependencies can make it harder to change anything because a large number of related changes needs to be made (high operating cost); these changes take time (lower agility); and changing anything risks breaking something unexpectedly (high operational risk). That’s why reducing dependencies to the minimum possible makes such good sense. SOA is therefore just common sense.

Why didn’t SOA succeed then to the extent that it should have?

The IT industry has seen SOA as just a bunch of cool technology products that can reduce the tight coupling between systems, and has forgotten about an equally important source of dependency. That’s the applications themselves, or, boiled down to the essence, it’s the nature of the data that’s exchanged between systems that is responsible for much of this tight coupling.

There is both a technology layer and a data layer in SOA, and we in the industry have been so fixated on the technology layer, so dazzled by the cool products we have, that we haven’t paid attention to what we do at the level of business applications and the kinds of data they exchange.

SOA, then and now
Again, mind you, I’m not knocking the significant advances made at the technology layer to reduce dependencies between systems. There was a time when data encoding was proprietary, and no one had standardized on the ASCII format that we take for granted now. Today, encoding standards like XML and JSON have taken that standardization a level higher up the stack, so systems can not only exchange data, but even make sense of its structure in a uniform way.

Today, DHCP and wireless computing have made the process of getting onto a network completely transparent. That’s what the standardization of protocols has done, and when we talk of SOAP and REST, that’s just an extension of protocol standardization further up the stack.

We even have standardization of “message exchange patterns” and “qualities of service,” so systems can agree on highly complex and sophisticated aspects of communication without having to invent them.

All of this is the contribution of the SOA technology layer, so it is very significant.

Today, ESBs physically mediate between systems, and registries do the same logically through lookups, so one would expect that everything should be loosely coupled, and voilà! We have SOA. But we don’t, do we? The proof in the pudding is in the failure to achieve lower operating costs, improved business agility and lower operational risk.

Which brings me to data.

Applications still have implicit dependencies on each other. There’s data that’s assumed knowledge, tucked away inside someone’s head or documented on paper somewhere, or worse, not documented at all and forgotten. Implicit dependencies trip you up when something changes, because these are the things you forget. That’s where operational risk comes from.

Legacy or unnecessary dependencies continue to exist past their legitimacy. These slow you down when you have to make changes, because changes are not as simple as they could be. That’s where reduced agility comes from.

Both of these also increase the cost of making changes. That’s where operating costs come from. It’s also why SOA seems such a failure despite the advances made at the technology layer. We’ve goofed up in our application designs!

So this is what we need to do at the data layer:
1. Identify all implicit dependencies and make them explicit. Make sure every data-related dependency between two systems is captured in an explicit contract between them.
2. Remove unnecessary dependencies. This is easier said than done, because it’s hard to get funding for cleanups that don’t deliver a business benefit, but be opportunistic about it. Clean out the crud whenever you’re tasked with making related changes.
3. Ensure that internal (domain) data and external (message) data cannot be derived or generated from each other. Changes to the internals of a system should not impact other systems.
4. Settle on a pragmatic and sensible data model for service messages. Services should use a consistent vocabulary, but chasing after a single Canonical Data Model for the entire organization is a waste of time and resources. Look for smaller data models for discrete domains within the enterprise.

If we can ensure loose coupling at the data layer as well as the technology layer, it means we have understood SOA. We’re then mature enough to take our business into the cloud, and can possibly derive the benefits of multi-tenancy, elasticity, pay-as-you-grow, and other goodies—but not until then.

Ganesh Prasad is a consulting architect at WSO2. He blogs at wisdomofganesh.blogspot.com.