SignalFx, a SaaS to Monitor Apps at Any Scale
Internet-scale companies that invent their own tools is a topic of of interest as more of the world builds out their own distributed architectures and looks for ways others before them have tread. The people who developed those tools are quite accomplished. As is well documented, many have used their skills and experience to create new tools and companies to support them.
SignalFx, which today is announcing $20 million in funding from Charles River Ventures, is a SaaS provider for monitoring modern applications. It had previously raised an $8 million round from Andreessen Horowitz. Andreessen also participated in this latest round. The roots of the technology they have developed stem in part from the work SignalFx Co-Founder Phillip Liu did as a Facebook software engineer working on a team that built systems management tools for the rapidly growing social network. In those days, much of Facebook’s experience came from developing their own tools to manage the ever growing scope of its distributed architecture. The problems they encountered are similar to the problems many companies face today.
Early on, Facebook used open source tools like Nagios and Ganglia that were designed for single server instance polls. As the number of the people visiting the site jumped, the Facebook team’s job became more complex. They would often find themselves asking: “Is the port available? Can I ping a particular server?”
It became harder to configure as Facebook scaled. It was hard to keep the noise out. The configurations had to be kept in sync, but there was always a window when that synchronization could not be achieved.
In distributed environments, the failure of one particular server may not mean disaster. But older, traditional monitoring environments are not built to adapt to the nuances and effects on performance when one node in a cluster fails. Those older systems detect a failure and sound the alarm. That’s often a false alarm, because if there are 100 nodes and one goes down, the system is not at risk of failure. End users will often never know there is a problem at all. So sounding an alarm every time a node fails just does not make any sense.
SignalFx aggregates the data from across a distributed architecture, monitors the trends, identifies patterns and then notifies users when things are abnormal. The data streaming engine can scale to billions, even trillions of metrics per day. It’s not too much of a surprise, considering that modern apps tend to be distributed with dozens of components.
SignalFlow is the the company’s data streaming service. It’s an analytics pipeline that provides insights within seconds of the data coming in. Customers, including Yelp, write to the SignalFx API to get the rich analytics capabilities that come with the service.
The customers are often developers, the people who increasingly do operations and as a result are more in charge, Co-Founder Karthik Rau said in a phone interview yesterday.
More companies are moving from monolithic applications to microservices, and with that power comes responsibility that developers have to carry.
These developers deploy apps and can immediately see how the services are performing, Rau said. They want that immediate feedback about inefficiencies or how the API is getting taxed.
It’s not like the old days that called for the operations team to closely monitor how things were performing on-premise. Today it’s not downtime but latency, which is always needing to be optimized. That work often goes to the developer who is monitoring the app to make sure it is performing well.
SignalFx has a lot of competition to face. It’s a crowded space, but Liu and Rau believe they have a platform that can stream monitoring data and perform analytics like no one else.
That may be true. But there are also a sea of open source projects that are continuing to evolve. And there are companies such as App Dynamics and New Relic that offer app monitoring. Datadog, Librato and Scout are a few of the others.
In the meantime, SignalFx is another example of the deep impact Internet-scale companies are having as companies all over the world build out their own distributed infrastructure.
Feature image via Flickr Creative Commons.
SignalFx and Datadog are sponsors of The New Stack.
