How SigOpt Wants to Optimize Your Machine Learning Algorithms
Powering recommendation engines, speech recognition and way-finding apps on our smartphones, machine learning permeates our lives in more ways than we realize. The machine learning models behind these systems, however, can be extremely difficult and non-intuitive to optimize. Experts don’t always know why exactly some model configurations work in certain cases and not in others, and even the simplest models can have a dizzying number of parameters to tune. Often these models are tuned using trial and error, but this can be time-consuming and costly.
To tackle this very real problem, San Francisco-based startup SigOpt offers an optimization-as-a-service API that allows users to seamlessly tune the configuration parameters in artificial intelligence and machine learning models. It’s a subscription service based on how many models a user tune per month, and recently raised $6.6 million in Series A funding last year. SigOpt optimizes models (and even advertising and products) through a combination of Bayesian and global optimization algorithms, which allows users and companies to boost the performance of their models, while cutting costs and saving the time that might have been previously spent on more tedious, conventional tuning methods.
Some of the advantages of SigOpt’s service include its ease of use: it can be easily integrated on top of any existing workflow, and its graphical interface also simplifies the process. It also uses what the company calls “black box optimization” — meaning the API doesn’t need access to the underlying data to optimize it, meaning that companies’ ML models and intellectual property are kept private. SigOpt also allows users to tune any algorithm in parallel by performing as a “distributed scheduler” for parameter tuning, so that serially arranged models can be tuned in parallel, so that better results are achieved faster across a larger scale. So far, SigOpt has been used in both academic and enterprise situations, from fraud detection to research and product development. We asked SigOpt co-founder and CEO Scott Clark to help us better understand how SigOpt came about and how it works under the hood.
What prompted you to start this company, and what real-world problem(s) are you looking to solve?
SigOpt was born out of the desire to make experts more efficient. While I was completing my PhD [in applied mathematics] at Cornell I noticed that often the final stage of research in fields as diverse as bioinformatics and computational fluid dynamics was a domain expert tweaking what they had built via trial and error in the hopes of achieving better performance. Because more sophisticated techniques were often unavailable or unusable, often this tuning was performed via expensive brute force, randomized search, or manual tuning.
After completing my PhD, I developed an open source package, the Metric Optimization Engine (MOE), while at Yelp to solve this problem, and used it to optimize machine learning models and A/B tests at Yelp. I found that many experts in the industry were facing the same challenges that my colleagues were in graduate school, needing to tune their recommendation systems, search algorithms, fraud detection methods, algorithmic trading strategies, and more. We built SigOpt to allow these experts to use the best possible techniques for solving their optimization problems via a simple API so they could focus building what they love.
What is “black box optimization”?
Black box optimization is a form of optimization where you can only observe the inputs and outputs of a particular system and nothing else. The goal is to find the input configuration that maximizes an output or set of outputs. This is a particularly difficult form of optimization because you have no way to introspect the underlying system to help decide what to try next, you can only observe the raw output of the input you put in. An example is the hyperparameters, architectures, and data pre-processing parameters in deep learning pipelines. These are the configuration parameters that need to be set before the system does any learning at all. The output of the system is only observed after the training is complete, which can be very time-consuming and expensive.
Describe how the technology behind SigOpt works in more technical detail.
SigOpt uses a variety of methods from Bayesian and global optimization. This field is sometimes called “optimal learning.” The goal is to efficiently sample a time-consuming and expensive-to-evaluate system, with the goal of maximizing some output. These techniques try to trade off “exploration” and “exploitation.”
“If you are not tuning your model configuration parameters you are leaving money on the table.”
Exploration is trying to learn as much as possible about the underlying system and how individual parameters interact to produce the output. Exploitation is trying to leverage this information to increase the output by as much as possible. By trading off these two approaches we can find globally good configurations by learning from previous attempts.
What are some examples of SigOpt’s technology in action?
SigOpt has helped experts tune their models in both industry and academia around the world. Examples include technology companies like Hotwire that use SigOpt to tune their search ranking models. Algorithmic trading firms use SigOpt to tune their underlying strategies. Researchers in academia have used SigOpt to tune models related to material science (finding the next great battery), bioinformatics (building CNNs for DNA sequencing), public policy (text data analytics for healthcare), and more.
How might SigOpt’s technology be integrated into a company’s workflow?
SigOpt bolts on top of any existing ML or AI workflow. A user provides SigOpt with information about the configuration parameters in their system, like hyperparameters, architecture, or pre-processing parameters in a deep learning pipeline. SigOpt suggests specific configurations for the user to try, within their current infrastructure. The user reports back how well these configurations performed on a given objective (like accuracy or expected revenue). SigOpt learns from this information to suggest the next best configurations to attempt.
This process is repeated as we converge to the best configurations that maximize the objective of the system. Because SigOpt is a black box optimization service we don’t need to access the underlying data, model or validation strategy to perform optimization. The SigOpt REST API needs only the configuration parameters and how well they have performed. This allows us to bolt on top of any underlying system, whether it is proprietary or not, and your data remains safe within your current infrastructure.
What distinguishes SigOpt from other comparable services?
SigOpt is the only commercial provider of black box optimization-as-a-service. We are different from services like Google’s Hyperparameter Tuning because we work on any underlying ML or AI model and pipeline and can tune any objective instead of being locked into the Google Cloud Platform, Tensorflow, and specific use cases. We are different than open source methods because we are easier to set up, use, administer, and we provide better results, faster [Author’s note: Check the peer-reviewed paper from the 2016 International Conference on Machine Learning with more details].
SigOpt outperforms traditional approaches to this problem like brute force, random, or manual search by being orders of magnitude more efficient. We offer a complimentary trial of our SaaS API subscription to prove this and are happy to be compared to any comparable service (whether internal or external).
Why should companies consider using such an optimization service?
If you are not tuning your model configuration parameters you are leaving money on the table. If you are tuning your models using standard techniques like grid, random, or manual search you are wasting expensive resources and getting poor performance. If you are trying to build or modify open source algorithms you are wasting valuable resources solving a problem outside of your line of business that could otherwise be used to differentiate your product. SigOpt provides a simple API that gives you access to an ensemble of state-of-the-art methods administered by world experts.
To find out more, check out the SigOpt demo, and read the developer documentation.
Google is a sponsor of The New Stack.
Feature image by Jörg Buntrock via Unsplash.