Choreo, WSO2’s New iPaaS Built on Top of Ballerina
Prior to the emergence of cloud computing, enterprise integration projects were either internal, serviced through an on-premises middleware platform, or external business to business (B2B) projects generally serviced through Electronic Data Interchange gateways. Increasingly though, many enterprises need to integrate data from a wider variety of sources, including IoT devices and mobile apps, causing a proliferation of APIs developers need to work with.
In addition, modern cloud architectures and Kubernetes, particularly when compared to earlier PaaS models such as Heroku, suffer from burgeoning complexity, prompting many companies to develop their own internal developer platforms.
At the same time, a shortage of skilled programmers means that businesses are struggling to recruit professional programmers to support ever more ambitious projects. This situation hampers IT leaders’ ability to deliver business automation in a rapid and reliable fashion.
An iPaaS provides capabilities to help integrate any combination of cloud and on-premises endpoints, including APIs and IoT devices. Key vendors in the space include Boomi, Informatica, Jitterbit, Microsoft (Azure Integration Services), MuleSoft, SnapLogic and Workato.
In June, a new integration platform entered the field, with features that are designed to help bridge the gap between low code and experienced developers: WSO2 has announced Choreo, currently in public beta, a low-code iPaaS for Kubernetes, built on top of the company’s open source programming language, Ballerina.
Round-Tripping Between Low Code and Pro Code
Increasingly, low-code application platforms, or LCAPs, have attempted to give business users without strong programming skills tools to help create apps. The idea is to help companies innovate faster and free up skilled developers’ time for tasks only they can do. A recent Gartner report claims that, by 2023, “over 50% of medium to large enterprises will have adopted an LCAP as one of their strategic application platforms.”
A long-standing issue with low-code platforms is that, while they can allow non-programmers to build applications, they tend to be a particularly egregious example of what Stack Overflow founder Joel Spolsky refers to as “leaky abstractions,” meaning that the implementation details tend to leak through.
A consequence of this is that, at some point, a professional developer must tinker with the code the tool generates in order to fix it. Skilled developers have their own tools, and won’t necessarily want to learn new ones in order to help repair something that a business user has written. Thus the visual low-code environment must be able to both reflect and cope with changes made to the underlying application code. If it can’t, subsequent changes in the low-code environment will undo changes made to the underlying source code, or may introduce bugs into the service.
“The difference between this era of low-code software and the attempts that precede it (4GL, RAD, etc.) is that business users and software developers can jointly contribute to the solution,” Doug Hudgeon, CEO of Managed Functions and a low-code expert wrote in a message to The New Stack.
To support this collaboration, being able to easily round-trip between the visual development environment and the underlying source code is central to the design of Choreo. To achieve it Choreo uses Ballerina, its language designed for writing network-distributed applications.
“I’m a fan of Ballerina,” Hudgeon told The New Stack. “It is in the sweet spot where it is usable by business users but also fits into standard enterprise software deployment workflows.”
Ballerina is still a young language, of course, with the 1.0 release having arrived in Sept. 2019. Since the 1.0 release there has been another major release — Ballerina Swan Lake — currently in beta. This is the version that Choreo uses under the hood.
Unsurprisingly given its youth Ballerina adoption is still quite limited. “We have 1,200 users on our Slack channel, and over 500 GitHub repos that actually use Ballerina in their code,” said Samudra Weerasinghe, campaign specialist at WSO2.
The graphical representation in Choreo’s low-code editor uses the semantics of sequence diagrams and flow charts to represent the program code. To make the round-tripping work, it uses a graphical representation of the actual Ballerina syntax tree, meaning that there are no translations between the visualization and the code that runs on the platform. This allows a developer familiar with Ballerina to make changes to the code, without causing issues for somebody working through the low-code editor who doesn’t understand the programming model.
Rendering is handled via the Language Server Protocol, an open, JSON-RPC-based protocol, originally developed by Microsoft, to standardize and provide required additional metadata for language tools. In Choreo, the low-code editor and the text-based “pro-code” editor continuously communicate with the Ballerina language server using LSP, based on the edits that happen on either end, to render the diagram or generate the code.
“The reason that this rendering is possible is because every Ballerina construct in the language has a visual representation,” said Srinath Perera, senior vice president and chief architect at WSO2.
In addition, extensions added to Choreo will automatically be supported by the low-code editor. For example, the connectors developed using Ballerina and exposed with OpenAPIs will render without making any user interface changes to the editor.
Deployment
The Choreo platform stores the generated source file in a private Git repo associated with the user account. Users can clone the repo, edit the Ballerina code using Microsoft VS Code, commit their changes, and merge to the same repo. Choreo will pick up the updated code, display it in the graphical and text editors, and use it with the build pipeline.
Regardless of whether low code, full code or a combination of the two is used, Choreo provides a one-click deployment method to Kubernetes.
Currently, the full DevOps experience isn’t publicly available, but “we run a full CI/CD pipeline, and our goal is to give the full DevOps experience with the pipeline,” Perera said.
Monitoring
By default, Choreo collects telemetry data without the developer needing to add any instrumentation, including application- and container-level logs, alongside CPU, memory, and other metrics from the Kubernetes pods and machines where the application is running. The Ballerina language also has built-in tracing and telemetry collection, which can provide activity and performance data at the statement level.
The data generated is pushed through an OpenTelemetry spec compatible GRPC-based protocol to a message queue, and then saved to a NoSQL database.
The observability view presents the low-code diagram of a Choreo application with annotations about the success rates of different paths and the time taken by different statements in the system. An additional panel shows details on throughput, latency data, and logs.
As with any distributed system, support for monitoring is particularly important. As one API analyst, who asked to have their name withheld, wrote, “the danger with these LCAP tools is the loss of visibility (they don’t typically fit into existing monitor/perf ecosystems). LCAPs have the potential to become the new ‘shadow IT,’ and that is a big danger.”
Although the use of OpenTelemetry under the covers should in theory make it possible to send the data to enterprise-standard monitoring tools that support the emerging standard — such as Dynatrace, Honeycomb, Lightstep, Microsoft (Azure Monitor), New Relic and Splunk — the usefulness of this isn’t yet clear. Said Perera, “You would have some level of interesting data, but not a lot of detail.”
Choreo is currently in beta on Azure, with general availability and support for other cloud platforms expected later this year.
