Why Microservices Succeed in App Integration Where Web Services Stumbled
Integrations aren’t sexy, but there’s a quiet revolution taking place in this enterprise market. The shift to microservices and reusable REST APIs is pushing changes at the very architectural level of enterprise infrastructure. When deployed properly, a REST API-based architecture can offer a world of on-demand integratable services and data. And when its coupled with the push for DevOps practices and procedures, the tearing down of group silos, and the emergence of new technologies such as GraphQL, large companies now have a lot of potential wins and ROI that can be juiced out of internal software architecture alone.
And yet, legacy systems are not going anywhere and they contain important applications and data that must interface with these new-fangled RESTful APIs. For enterprises, this means integrating these systems together; “integrations” is an almost innocuous buzzword that has morphed into something far different from what it meant back in the 90’s.
The biggest cosmetic differences between last decade’s advocacy of CORBA and SOAP for data integration and the use of REST today are the size of today’s services. In the past, monolithic services were part of the equation, but today they’re the enemy. Microservices are all about paring down an application like slicing up an apple. Instead of integrating existing services and data sources, this new world of microservices is about composing new applications out of the basic elements of an enterprise’s services architecture.
It’s a play that requires coordination at the requirements level, and the incremental winding down of legacy systems in favor of many small replacement systems.
Free the Dev
This work to bridge the different generations of IT has created new pockets of business for companies built around data and application integration. SnapLogic, for example, offers a customized way to build integrations for the business people in the room, eliminating the need for a developer when, say, connecting a new data source to an analytics platform.
James Markarian, chief technology officer of SnapLogic, knows the traditional integrations market well. He spent 15 years at Informatica as that integrations company’s Chief Technology Officer, and he’s putting that long track record to work at SnapLogic. He sees a large opportunity for integration platforms that can eliminate the developer from the equation.
“I like the universe metaphor,” said Markarian. “The integration landscape is enormous, and like the universe, it has no actual center. It’s impossible to find something that does everything in the integration space. There are small solar systems that compose the broader universe, and maybe each of those smaller systems has their own requirements.”
“If we extend the idea of the universe, data is a little bit like the universe in that entropy increases over time. What that means in the data or integration space is that it’s a combination of changing requirements and heterogeneity in terms of the data types you’re dealing with. Anything you look at — styles of integration, velocity, frequency — that all seems to be changing over time,” said Markarian.
He said that his time working on these types of problems in the 90’s taught him very firmly that interfaces won’t remain the same over time. He learned this while working with CORBA and Microsoft’s Distributed Object Component Model (DCOM). He distinguished between the interface you design, and the one you end up with after reworking it to live in the real world over time.
“Generally my observation was those systems imploded under their own weight,” Markarian recalled. “We discovered the time problem: even if you could come up with a set of interfaces at T=0, at T=1 and beyond you’d wind up for a reworked set of interfaces.”
The finer grain the services, middleware, or database, the less likely they are to need significant rework or versioning as time goes on, the SnapLogic CTO argued.
“When you think about all the things an interface or a set of interfaces might have to do today, it’s actually maybe a very long list of things it has to do, and one of the theorems I kick around here is that there’s generally less than one effective interface author per organization. Most people don’t really know how to take all these things into account. The result of that would be that you have a lot of folks creating and using APIs that will necessarily need to go through a massive amount of work in the next iteration and haven’t thought through the evolution of that API,” said Markarian.
Markarian said that business people should be the front lines for integrations. They’re the ones who have the business need for a given integration, be it a data integration or application integration. The goal of SnapLogic, said Markarian, is to make integrations self-service for business users.
“The audience of non-technical users of something is necessarily bigger. That doesn’t make it lucrative, but that’s one more way of looking at things. I think if you looked at the irony of this is that when heavy IT roamed the earth in the 80s and 90s, you had many more technical people in central HQ. They tended to be more technical and less business focused. It was a better time for a more technical solution,” said Markarian.
Products not Pipelines
Uri Sarid, Chief Technology Officer of MuleSoft, sees the interface issue as one that can be solved by better tooling and specifications. He’s behind RAML, and recently backed the OpenAPI specification. The issue isn’t necessarily interfacing with those APIs, but rather, interfacing with them over time as they change and evolve. Therefore, said Sarid, it is useful to have tools that can programmatically regenerate those interfaces.
“I think it’s a pretty clear black and white line: I think the traditional integration approach has been to build pipes between applications. That’s what people have had visually in their minds, so when we say ‘we’re in integration,’ people think we build pipes,” said Sarid. “You do need to coordinate across applications, but by building pipes not only is it hard to get these things to work together, but it also cements them together so you end up being able to move less fast than you were before.”
“What you need to do is lean into the same principles, like in the DevOps movement. You carve up applications into services and link them to each other. It’s incremental. You end up abstracting capabilities out from your applications as APIs, then you integrate and compose them into higher level APIs. That more productized approach no longer feels like traditional integrations. It feels like a set of services and reusable components, which is what the business people and developers wanted. Business people like the idea of having a set of capabilities and being able to compose something new without having to figure out if this breaks anything,” Sarid said.
Sarid sees that modern agile business is able to bring developers and business workers together to align their goals. This is a new development, and something that is contributing to the driving forces behind DevOps said Sarid. Instead of asking business people to get their own data, Sarid envisions a world where existing applications are reinvented as microservices and allowing business users to compose an entire application from pieces. This level of abstraction, enabled by the use of RESTful APIs, makes it easier for developer and business person to build interfaces based entirely on the language of the business, instead of in the language of computers.
“I think this aligns developers with business people. It’s a different thing because it’s much longer lived. It’s no longer ‘Do you have a better pipe,’ but, ‘Are you re architecting your enterprise to be more agile?’ said Sarid. “We’ve found it almost refreshingly surprising, the notion of APIs, no matter how simplified the business side makes it, it’s something the business side very much understands these days. We’re hearing very business-oriented IT leaders talk about APIs as a set of capabilities. Now they’re couching it in the language of APIs. There is a natural alignment point around APIs.”
The future of enterprise integrations might just have room for both approaches, however. Sarid believes we are still in the early days of dealing with and processing RESTful Web services, and he said that he expects GraphQL, an emerging rich query language for APIs, to attract the interest to enterprises going forward.
“There are two things GraphQL has picked up on,” said Sarid. “In the current REST approach, there don’t seem to be good solutions. Ultimately, the client applications should be catered to. It should be very possible for anyone writing a client, like mobile, to control the shape and amount of data it gets from the server. Today’s RESTful APIs don’t give you a lot of that control. The second is that if you don’t want to assemble that data yourself, you end up making lots of calls, and on mobile, that chattiness comes at a high price. GraphQL solves these problems. It allows the client to query for data, and made a very compelling packager of it, like Docker packaged containers.”
Markarian, who was for a short time working in the venture capital world, was looking at investing in a GraphQL company before joining SnapLogic. “I like the idea that they’re attempting to provide the composability of SQL with APIs, but there are a couple of issues. APIs aren’t, by nature, implemented in such a way that they can respond to query predicates by pushing out certain elements. In terms of post processing data that comes out as a new transient interface.”
“I think it has potential, but there’s still a question about the audience. Who would be using an interface composition like GraphQL? Does it approach more of the SQL crowd? That starts getting into business, but this is still a fairly low level. The technology is not exactly SQL, and generally things that aren’t exactly SQL,” aren’t terribly successful, said Markarian.
Sarid admitted there are still questions in the GraphQL space when it comes to infrastructure. “In doing that, they said; ‘forget all the RESTful, normal HTTP, stuff does it this way,’ and as a result, there’s a lot of challenges. You don’t know how to cache these things. You have to implement new APIs.”
MuleSoft has been working to bring GraphQL closer to RESTful way of doing things that everyone is familiar with.”We have an approach where we’re working on leveraging existing standards like hypermedia and RESTful APIs, to still allow you to create queries across these APIs in a non-chatty way without having the producers of these APIs have to implement GraphQL endpoints,” said Sarid.
Feature image via Pixabay.
