How Serverless Platforms Could Power an Event-Driven AI Pipeline
This article is a post in a series on bringing continuous integration and delivery (CI/CD) practices to machine learning. Check back to The New Stack for future installments.
Event-driven artificial intelligence (AI) can lead to faster and smarter decisions, but will require a reorientation of organizational processes and team composition, alongside a hybrid architecture structure that takes the best of serverless and combines it with stateful database stores.
Orit Nissan-Messing, co-founder and chief architect at serverless platform provider Iguazio, says that event-driven AI can lead to immediate insights by reducing the time to under a second between when an event occurs and the data needed to drive action. It can refresh data models with external data in realtime. And it can leverage the cloud to maintain agility through continuous development and deployment practices.
Serverless can be ideal to create this realtime AI pipeline, she pointed out, but does have some shortfalls, notably a slow performance and lack of concurrency; and challenges in managing complex functions and multiple event data sources.
If these challenges can be overcome, serverless architectures could help build AI pipelines that take automation to a new level. Right now, serverless workflows typically compose APIs, functions and events to automate processes and take repetitive actions (while logging everything that happens). In the future with an AI pipeline, that automation can become more sophisticated. Instead of just triggering events in a serverless workflow, it will be possible for workflows to also simultaneously refer back to every previous instance of running that workflow and see when the optimal outcomes were achieved. It can then choose actions that will repeat that success.
Immediate Use Cases for AI in Serverless Architecture
AI pipelines can help improve performance of typical business workflows by drawing on historical data and context, and use that data model to take positive action.
Some examples include:
- Displaying data in location-based apps: Location-based apps need to take a GPS signal of the consumer, understand their previous purchasing behavior, know what other customers with similar profiles have been purchasing, and then serve up relevant content to help that consumer make shopping decisions. It may also be necessary to know how often you can make suggestions to the customer before they switch off notifications.
- Risk management: When customers buy in bulk from a business, either in a physical product that needs to be shipped, or are racking up usage fees of a service, it is useful to be able to do a risk assessment of the customer’s capacity to pay their bill or monthly account. Knowing when to allow a customer to increase their usual monthly spend or to hold back until payments are forthcoming can be predicted with an AI data model.
- Infrastructure monitoring: In IT, or physical processes like in manufacturing or managing transport and logistics, it is useful to see where routes are congested so traffic can be pushed through an alternative route. That may also involve reviewing data on who is impacted by slow arrivals from traffic delays so that a business can make trade-off decisions, based on SLAs or customer value, on which customer segment should be prioritized next.
- Personalized content: Whether through displaying recommended content or using personal and segmented consumer behavior, a business can make decisions on what content to display to an individual reader when they visit a single page app site.
All of these cases are drawing from potentially thousands of data points from previous times these actions have been taken, and in serverless, the data model can be reviewed and decisions made instantly — reacting in under a quarter of a second, for example.
Nissan-Messing says an AI pipeline in serverless goes through four stages:
- Transform: First data is ingested from realtime sources and needs to be transformed into a common format so that it can be added to models along with other available data, which can include the extraction of meaningful data from text or voice using natural language processing.
- Enrich: This realtime data is then contextualized by bringing in other data sources such as historical records of previous instances or additional environmental data, like weather or traffic conditions..
- Infer: The serverless pipeline process is now able to draw on a machine learning model to predict the optimal outcome based on the realtime data and the contextual enrichment.
- Act: The serverless pipeline can now trigger the next action to take based on the probability of a successful outcome.
But there are actually a few steps that need to be in place before being able to build this workflow. For starters, the business must have a data science organizational infrastructure in place.
Creating an Organization That Supports Data Science
Telco and payments platform Africa’s Talking provides a number of open APIs to developers in order to manage payments, buy airtime, and integrate both voice over internet and messaging into applications and services. Although the organization does not yet use serverless technologies for its growing AI operations, the technology is in the mix.
Currently, the platform infrastructure uses a combination of Mesos and Kubernetes to manage containers and simplify application deployments. This allows the company to experiment with workflows where developers can write, package and deploy code without having to directly interact with the server. For the company, getting started with AI pipelines first meant becoming a data-oriented business. Africa’s Talking Lead Data Scientist Kathleen Siminyu has been managing this activity.
“We are very analytics heavy at the moment,” Siminyu said. “Data analytics is what the current infrastructure can do, so it is the low-hanging fruit. And to enable better data-driven decision making, we want data science staff to sit in product teams. So we have a data scientist who deals with messaging, one who deals with payments and airtime, and one who deals with voice,” she said, pointing to the three main product streams of the business.
“Setting up our data scientists in product teams is something we learnt from experience: originally, our business was organized in more conventional teams like HR, and finance business units, but we have seen value in encouraging people to be really good at one thing. So when we are running a product, we want you to be the data person who is expert for that product team. Then in addition to that, I’m the person who looks at the tools in the serverless, data science and big data ecosystem and co-ordinates with the data engineering and infrastructure teams to make decisions around which ones the teams use,” Siminyu said. “My strength is in Spark, but that has required figuring out how Spark works with Kubernetes, and where the logs go. I have ended up being pulled into a lot more discussions around infrastructure.”
But as the lead in the data science team, that makes sense and allows her team members to focus more on mapping the data models for each product they are working with.
“In the product teams, the data scientists are then working on mapping the things we want to do when the infrastructure is at maturity,” explained Siminyu. “Infra does not allow you to have a good overview of the problem. One of the biggest problems is that there are many different services that have data. The robustness of the infra influences how agile you are in pulling data together, and how you can enrich insights from your data.”
Without the ability to pull in multiple data sources, a data model risks repeating poor business decisions of the past. No one is immune: most recently Amazon’s face recognition AI model has been shown to misgender some faces 38 percent of the time. Those kinds of inaccuracies are often due to too much data coming from the one source, rather than diversifying the data sources that help train the model.
To avoid this kind of error in the data modeling at Africa’s Talking, Siminyu is already considering some new approaches, such as establishing an internal ethics review committee.
One of the strengths of a serverless AI pipeline could be that it is primed as an infrastructure approach to be able to draw on a more diverse range of data sources and include them when triggering decision events.
Overcoming Serverless Limitations in AI
At Iguazio, Nissan-Messing and co-founder Eran Duchan were instrumental created the open source serverless platform, Nuclio.
“There are multiple challenges we are solving with Nuclio,” Nissan-Messing said. The software ingests events from many different event sources, including Pub/Sub streams, HTTP, MQTT, Azure event hub and AWS Kinesis.
“With Iguazio, we are then able to provide a distributed application and database layer that can treat data very fast, with non-blocking invocations, so that we can process many events in parallel in the same process, overcoming the lack of concurrency downfall of most serverless AI pipeline approaches,” said Nissan-Messing. “This gives a huge boost to the number of processes we run, and with serverless the compute can autoscale. We have also created a system that allows you to not only have some state, but to keep the connection to a database open so that it reduces the start/stop nature of serverless which can slow down performance.”
Nissan-Messing points out that after each runthrough of the workflow, the data should be logged and then added to the training model, but that that compute process is best done in a more conventional setup.
“Training data models in serverless isn’t the best idea,” warns Nissan-Messing. “There are existing machine learning frameworks that are very strong. We have customers who update their models weekly, others who add new data to their models less frequently, it depends on the customer use case. The cost of training the model takes a lot of computational so you don’t want to do it too frequently unless it is worth it, because training the model takes a couple of hours.”
Successfully using serverless for AI pipelines will require the combination of a number of processes. There is the infrastructure pipeline that transforms, enriches, infers and acts. But before that is the ability to compute the training model away from a serverless infrastructure. And before that is designing a data model that allows for a diverse range of sources to be drawn in. And before even that, there is the need to set up organizational processes and team composition to best prepare for data-enriched decision-making.
Images in this article are drawn from Orit Nissan-Messing’s presentation Serverless and AI, presented at Serverless NYC in October 2018
