Ali Ghodsi, CEO and co-founder at Databricks, has been a major influence in three big open source projects — Apache Mesos, Apache Spark and Apache Hadoop. What’s interesting about all of these three projects is that there are very successful companies around these projects. We sat down with Ghodsi for this latest episode of The New Stack to talk about these Open Source projects and machine learning.
According to Ghodsi, machine learning replaces manual, repeatable processes. Older systems were rule-based which would bring some level of automation, but they had their own limitation. First of all, any rules-based systems can be easily gamed. Second, you can’t really use such a system to monitor and respond in situations where billions of people are interacting with each other in real time.
A chat program, which is used by teenagers can be a great example. You may want to monitor it for foul language or any alarming content; it can’t be done with rules and teenagers know how to bypass things. “Machine learning overcomes that problem and becomes a very powerful tool in such use-cases,” said Ghodsi.
Machine learning is also being used in medical science. It can help patients with Type 2 diabetes. Put a patient’s medical records with Genome together and machine learning can help in the creation of better drugs. It can help with credit cards fraud or protect your IT infrastructure from any intrusion.
In this interview, Ghodsi shared all the immense possibilities that are there with machine learning. Ghodsi is also an adjunct professor at University of California Berkeley and he is seeing a new generation of computer scientists that are looking at things from a totally different perspective.” They will actually think of things that you and I are not smart enough to think of. The next generation will come up with even better technologies and ideas,” said Ghodsi.
In This Edition:
1:29: How much day-to-day involvement does Databricks have with Apache Spark?
4:56: What problem is Databricks solving for its customers?
6:43: Who are some of your use cases?
9:24: The definition of machine learning and some examples of it
11:20: How can machine learning help the stack?
12:55: Exploring machine learning’s massive role in the years ahead.