API Management / Technology

DataFire Links Back-End Developer APIs to Create Data Flows Between Services

1 May 2017 9:00am, by

As software-as-a-service offerings increasingly aid developers with repetitive back-end work, the problem of siloed data rears its ugly head.

DataFire, an open source framework for processing and syncing data between APIs, databases and other applications, is tackling that problem.

It’s the brainstorm of brothers Andrew Brennan and Bobby Brennan, creators of LucyBot, which uses cookbooks to translate API requests into different programming languages for documentations and tutorials.

With LucyBot, they worked with open source video platform Kaltura to document more than 600 operations and create tutorials for more than a dozen common use cases. And AnyAPI used LucyBot to create documentation for every API in the APIs.guru directory.

With DataFire, the Brennans want to transition from serving API providers to serving API consumers.

“You’ve seen all these [SaaS} services spring up over the past 10 years or so to help developers do everything from tracking users, sending email, maintaining sales leads, things like that.

“Rather than writing your own email service, you can use MailChimp or SendGrid. Rather than having to write your own event-tracking software, you can use Google Analytics or MixPanel. That’s been a huge boost to developer productivity. Rather than doing all these boilerplate backend services, they can just integrate these SaaS providers and focus on shipping features unique to their product,” explained Andrew Brennan.

Without those offerings, developers spent a lot of time building things that didn’t help differentiate their product.

“Now even a small startup will be using a half-dozen different SaaS providers for tasks that are outside their core competence. The problem is that they now have this very valuable, actionable data siloed in all these different services. You have a bunch of insight on how people are using your website in MixPanel, you have their email addresses in MailChimp, their contact info and all that in Salesforce. The question now is how do you coordinate all that different info? That’s where we want to be able to help.”

They describe DataFire as Grunt for APIs or Zapier for the command line. It allows users to create “dataflows” to connect services such as GitHub, Salesforce and Slack, then sync, monitor and interact with that data. Dataflows can run on-demand or be scheduled and shared with others.

“While platforms like Zapier and IFTTT provide beautiful GUIs for building and sharing simple integrations, they often lack the power and expressiveness of good old-fashioned code. DataFire sought to fill this gap by allowing users to write snippets of JavaScript that control how data is retrieved and transformed,” Bobby Brennan wrote in a blog post at Hackernoon.

He listed three things existing products lack:

  • Ability to write code. There are some tasks that only code can solve, he said.
  • Ability to call three or more APIs in one flow.
  • Ability to schedule or run on demand.

Dataflows can be set up, for instance, to deliver news headlines from myriad sources to your inbox, sync GitHub issues to Trello or set up automatic retweets on Twitter based on sentiment analysis.

Bobby Brennan set up a dataflow he gets each morning with key statistics for the business such as page views and sessions from Google Analytics and number of new users from MongoDB. It’s also easy to set up monitoring alerts for database or other infrastructure issues, he wrote in a post at Medium.

In his API Evangalist blog, Kin Lane lauded Datafire:

“DataFire is new, so it has a lot of maturing to do as an API framework, but it has EVERYTHING that iPaaS [integration-platform-as-a-service] solutions should have at its core in my opinion. It’s API definition-driven, its open source, and there is a cloud version that any non-developer user can put to use,” he said.

“…DataFire is the iPaaS solution I’ve been waiting to see emerge and will be investing time into learning more about how it works, developing integrations and flows, and telling stories about what others are doing with it.”

The brothers open-sourced the integration framework in November and have been working to build a community around the logic of the back end, Andrew Brennan said.

“For an integration framework, it’s really important to have a large community of contributors. We provide connections to more than 300 API providers and we’ve barely scratched the surface on what’s available out there,” he said.

“Another reason it needs to be open source is that there are a lot of APIs that are just boilerplate across organizations. They just need to get email addresses from MailChimp into Salesforce. Or events out of MixPanel into Salesforce. These data flows are easy to share across organizations. That’s something we want to see more of.”

In the open-sourcing announcement, Bobby Brennan pointed to these features:

  • It provides access to the entire NodeJS ecosystem, including every package in npm.
  • You can easily factor your code into multiple files and share code across projects.
  • Flows can live inside any version control system, and you can work in the editor of your choice.
  • And flows have been decoupled from the DataFire platform. While you can still run and manage flows on datafire.io, you can also run them locally, or on AWS Lambda, Google Cloud, and Microsoft Azure via the Serverless framework.

Feature Image: “NASA-Led Campaign Studies Hawaii’s Iconic Volcanoes” by NASA’s Marshall Space Flight Center, licensed under CC BY-SA 2.0.

A newsletter digest of the week’s most important stories & analyses.