At AWS re:Invent conference last week, Cloud Elements launched Element Builder on the AWS Marketplace, with the aim of making a variety of API integrations available to developers as Lambda functions.
To date, Cloud Elements has been an integration platform focused on standardizing SaaS APIs in ways that make them interoperable for use in developer-created applications and workflows. Now, their new Element Builder tool aims to make these standardized API integrations available within the AWS Lambda platform, for any developers working with Amazon API Gateway, Mobile Hub or Serverless platforms.
“Cloud Elements addresses a key challenge by accelerating the publishing of SaaS application APIs as AWS Lambda functions,” said Tim Wagner, AWS general manager of Lambda and Amazon API Gateway, at the launch of the new service. “Developers using AWS are looking for ways to combine APIs from a variety of SaaS and software applications into their applications, to support their serverless architecture and scale their code without incurring additional costs to run.”
Until now, Cloud Elements has been focused on supporting developers to more easily integrate APIs without having to first abstract and normalize the different ways that individual APIs may manage data and service access, said Mark Geene, CEO of Cloud Elements.
Cloud Elements publishes integrations as API connectors and organizes them into API hubs, creating a standard across various SaaS APIs (for example, contact management), so that whatever cloud-based SaaS tools were being used to access data, it is now interoperable with other cloud-based SaaS tools. It is a similar model to how API management provider MuleSoft is creating an application network platform: integrations become less point-to-point and more one-to-many hubs that act as reusable and interoperable standards within an application development environment.
An Open API Distribution Platform
“We have now extended Element Builder so that you can not only create a connector for integration scenarios, but you can also distribute that connector as a microservice in the form of an AWS Lambda function,” said Geene.
“This is the first step in our strategy of creating an open API distribution platform. Amazon is the first of many platforms we will work with to distribute our catalog of integrations and publish them into marketplaces like AWS Amazon Marketplace and with other vendors,” said Geene, who also mentioned the Microsoft Azure, IBM Bluemix, and SAP HANA platforms.
The need for capabilities like Element Builder within the AWS platform reflects the growing convergence of microservices and APIs. After all, APIs are what make microservices architecture possible, and are giving rise to a new need for hybrid combinations that might take internal and proprietary APIs, mix them with third party service APIs or to connect data from SaaS tools, and with smaller microservice functionalities.
The New Normal: Stitching APIs, Microservices and Functions together
Recent API service and product providers have all looked to reorient their offerings to be more microservices-focused. API management providers like RedHat and 3scale have worked with external partners like Stormpath to introduce identity management as a microservice into their service offering. Nanoscale.io repositioned its serverless API architecture product to let users create not just APIs but microservices and functions for serverless computing.
Most recently API lifecycle tooling Stoplight.io has created a scenario builder aimed at helping automate workflows that might combine internal and third party APIs with AWS Lambda functions.
Marc MacLeod, founder and CEO of Stoplight.io gives the example of making low-level functions from the OpenCV node library for facial recognition available as lambdas. “It is about orchestrating the two worlds: the world of serverless and the lambda function by itself, and the world of third party and internal web APIs,” said MacLeod.
MacLeod explained: “Our general thesis is that as you get smaller and smaller individual functions, you can slice and dice them. APIs are not always easy to mix, so if their functionality is available in multiple smaller pieces, you can mix and match those smaller bits into a larger worker flow, which is what our Scenarios Builder helps you do.”
Cloud Elements is seeing this need by application developers to combine external APIs in their apps and workflows as microservices and functions, not as larger APIs, and their new Element Builder integration into AWS means that for SaaS already in their directory, those cloud products are now automatically available within AWS for integration by developers.
As microservices architecture becomes the paradigm of choice for large, distributed applications and network systems, APIs become not only more important but also need to be packaged up in a more granular fashion so that they are more composable as functional units that can be inserted into an application workflow.
“We are making our API integrations available through a publishing function,” said Geene. “You can take any API and have that available as a microservice to combine together in, for example, AWS Mobile Hub to create new application experiences.”
The move by Cloud Elements and other API stakeholders reflects three key trends seen across the API economy right now:
- The increasing growth of serverless as the new cloud and hybrid architectural format, overtaking DevOps discussions to bring in a wider range of developer interests.
- The need for APIs to be discoverable on multiple platforms.
- The as-yet unaddressed challenge of using machine learning and automation techniques to aid function, microservice and API service discoverability.
The Growth of Serverless and Widening of DevOps discussions
One comment Mark Geene made about AWS Re:Invent is that in the past two years he has seen it balloon out to be a key cloud technologies conference, spanning SaaS, application and integration developers as well as the more traditional focus of DevOps, which dominated the program two years ago. Now it seems that serverless is one of the key discussion topics, along with a widening of participation by developers involved in SaaS businesses and systems integrators. This reflects some of the major changes seen this year, where the pendulum is swinging back from an excitement around DevOps and back towards supporting developers to create new applications.
Serverless is a long way from being the application deployment and distribution solution for everyone — or even most — as businesses now face the need to balance on-premises data and network systems with cloud-based infrastructure. But examples like the recent move by SimilarWeb to solely cloud-based infrastructure as part of their growth management is not an uncommon story, and as that continues to happen for a wide range of businesses across various industry sectors, the interest in being able to manage applications completely in the cloud will continue to grow.
Publishing APIs on multiple platforms
With the growth of serverless application development platforms (AWS, IBM OpenWhisk, Microsoft Azure, Syncano, Stamplay, etc), API providers now need to be able to make sure their APIs are discoverable in a widening array of catalogs and marketplaces. This is in addition to optimizing their own developer portals for SEO, and interacting with developers in their domain of interest on StackOverflow. In the same way that an API provider takes on an additional workload by creating SDKs in multiple programming languages to support developer adoption, API providers must now also make sure their APIs are listed in the right way to be included in these various serverless and application platforms.
That has gotten even harder with the introduction of functions-as-a-service/serverless platforms. API providers must reorient their APIs into multiple functions and make them available with AWS as Lambdas for example. They need to format those same functions in a different way for developers with the Microsoft Azure platform. And with IBM OpenWhisk, etc.
“What we have are doing is targeting application providers: SaaS companies, and divisions of enterprises that are building digital business apps,” explained Geene. “There is a proliferation of various marketplaces where APIs are being consumed. The next generation of API consumption is going to happen on leading platform providers, and with Element Builder, we want to help providers get their APIs packaged up and into those various platforms.”
Solving Service Discovery
But even with the introduction of Element Builder and the widening of API tooling to encompass both microservices and smaller functions as composable units, there is still a key aspect of discoverability that is not only unsolved but actually worsened by the multiplication of granular services. Platforms like AWS don’t have catalogs of available lambdas. Developers need to know what resources and functions they want to stitch together into their application or build them themselves.
With the publication of API integrations as AWS lambdas, what needs to happen next is to have some sort of discoverability within application development platforms so that as developers begin to use objects like contact names or face images in their apps, the platform will notice what they are working on and make recommendations of available APIs, microservices and functions that might be relevant to their task (or object) at hand. That sort of discoverability is still some way away from being provided to developers. In the meantime, developers are left with searching through cumbersome catalogs or rebuilding their own functions because they are unaware someone else has already done so.
“This is the potential for the future,” said Geene. “This is where the market is going. Organizing how you can access all of these microservices is going to be more of a challenge for the future, especially as API and microservices get more granular.” Geene points to a complementary Cloud Elements product, Formula Builder, that lets developers create their own API using a “unit of logic,” a small code snippet that might act like a function. “Part of our vision of APIs and microservices working together is to let the customer choose and create that logic and to plug it into their application.” Geene says having Formula Builder and Element Builder connected is on their roadmap for the future.
This is where the machine learning opportunities will begin to integrate more with application design. Already algorithms are being integrated via APIs into workflows and application design. But machine learning to decide which microservice or function to use next in that work chain is still new and using existing application development platforms is still some way off. But publishing API integrations within AWS is a good place to start.