How AutoML Puts the Power of AI in the Hands of Business Analysts

Infusing artificial intelligence and machine learning into business applications is not a trivial task. Especially, when it comes to enterprises dealing with mission-critical applications, integrating ML with existing applications becomes a challenging exercise.

From the time an organization decides to incorporate machine learning to the actual deployment of a fully-trained model, there are multiple stages involving different teams and individuals with disparate skills. It has to go through the workflow depicted in the below graphic.

Click to enlarge

A Look at the Machine Learning Pipeline

Data acquisition involves identifying and extracting data from a variety of data sources such as RDBMS, NoSQL databases, data warehouses, third-party sources etc. Enterprises rely on advanced ETL (extract, transform, load) tools to aggregate data from heterogeneous sources.

Data exploration will provide insights into acquired datasets, and helps the data engineering teams to assess the quality of data. This phase will help the team in finding hidden patterns, correlations, missing data points in an aggregated dataset.

Data preparation phase deals with cleansing the dataset. Missing data points may get dropped, existing columns may get split, multiple columns may get combined, and finally, the dataset turns into a complete and a valuable input source for the remaining stages of the workflow.

Feature engineering is a critical step where data scientists identify dependent (label) and independent variables (feature) from the dataset. They perform techniques such as encoding, scaling, and normalization to make the data consistent and homogeneous. Feature engineering is critical to the success of an ML project.

The next phase, model selection, is all about choosing the right algorithm from the plethora of available statistical models. There may be more than one algorithm (sometimes even a dozen) to solve the same problem. An experienced data scientists will apply intuition combined with a deep understanding of the business problem to handpick a set of algorithms.

During the training phase, the fully-prepared, feature engineered dataset is used for training and testing the accuracy of the model. Input data is split into a training and testing dataset which is useful in assessing the accuracy and precision of the model.

Hyperparameter tuning involves tweaking various parameters of the training job to evolve an accurate model. This optimization is done for all the shortlisted algorithms used for training. This phase is the most complex of all the steps of the workflow.

The previous three phases — model selection, model training, and hyperparameter tuning — are iterated multiple times until the results are satisfactory. For each model chosen for training, hyperparameter tuning is applied iteratively.

Finally, the most optimal model is selected and deployed for real-time predictions. Developers may call the REST endpoint like any other API to generate predictions.

Enter AutoML

AutoML attempts to accelerate the process of evolving a trained model by automating the majority of the steps. Users are expected to upload the datasets and wait for the predictions to become available. From feature engineering to hyperparameter tuning, AutML automates the most complex steps of the pipeline.

AutoML creates a new class of “citizen data scientists” that puts the power of advanced ML directly in the hands of business users. It doesn’t mean an organization using an AutoML platform doesn’t need data scientists. They still have a role to play in bringing their experience and domain knowledge. But, AutoML makes data scientists more productive as repetitive steps in the model building process are automated, allowing them to use their unique expertise for optimizing the models.

When dealing with an AutoML platform, business analysts stay focused on the business problem instead of getting lost in the process and workflow. Most of the platforms prompt users to upload the dataset and then labeling the categories. After that, most of the steps involved in preparing the data, choosing the right algorithm, optimization and hyperparameter tuning are handled behind the scenes. After a while, the platform exposes a REST endpoint that can be used for predictions. This approach significantly changes the traditional workflow involved in training machine learning models.

Getting started with AutoML

AutoML is an evolving trend in the field of AI and ML. Public cloud providers such as IBM, Google, and Microsoft are pioneering the AutoML revolution by offering custom cognitive platforms. Enterprise players such as DataRobot are bringing the power of AutoML to businesses. Mainstream frameworks including Scikit-learn and Keras are embracing AutoML to simplify the workflow.

To get a feel of AutoML, refer to my tutorial on using AutoML vision on Google Cloud Platform.

AutoML perfectly fits in between cognitive APIs and custom ML platforms. It delivers the right level of customization without forcing the developers to go through the elaborate workflow. Unlike cognitive APIs that are often considered as black boxes, AutoML exposes the same degree of flexibility but with custom data combined with portability.

Some AutoML platforms such as Microsoft Azure also support exporting the fully trained model compatible with mobile devices running Android and iOS. Developers can quickly integrate the models with their mobile applications without having to learn the nuts and bolts of machine learning.

Microsoft has recently announced the availability of running cognitive services in containers. When AutoML models get exported into Docker containers, DevOps teams would be able to deploy them at scale for inferencing in production environments. They can host the containers in scalable clusters managed by Kubernetes and DC/OS.

With every platform vendor attempting to democratize machine learning, AutoML is evolving as the future of artificial intelligence. It puts the power of AI in the hands of business analysts and technology decision makers.

Photo by Anderson W Rangel on Unsplash.