Serverless Monitoring in the Age of Infrastructure-as-Code
Monitoring serverless systems requires a new level of acceptance of where infrastructure as code and production testing is taking software development in the age of distributed application architecture at scale.
“We are seeing a fast shift in the development of software,” said Or Weis, CEO and co-founder of serverless monitoring tool, Rookout. “Software now has thousands of instances, and a developer may never see any of them. A software developer may never know the environment specific parameters, for example, that an instance might have. So the level of ability to simulate how the code will run in a test environment is rapidly approaching zero.”
Monitoring is not as simple as just setting up alerts for performance, speed or time-outs but instead requires a higher level thinking that relates back to the intended goals of the application.
Weis said that means monitoring in serverless systems becomes about understanding observability on a much higher level. Serverless systems are creating an infrastructure-as-code mindset and practice, where an application holds all of the code for not just performing its functions and compute processes, but, because it is being run on serverless systems, is also the initiator of instances and scaling up when needed, automatically.
This means monitoring is not as simple as just setting up alerts for performance, speed or time-outs but instead requires a higher level of thinking that relates back to the intended goals of the application. This involves concepts such as infrastructure-as-code.
What does it really mean when we say infrastructure-as-code? “First it means you are trying to reduce the friction required to make software more agile,” Weis said. “Secondly, it means you need to work in sync with all of the layers of the software from infrastructure to application, so you need a layer of orchestration to manage all of that.”
Weis explains why we need a new approach to monitoring in this environment: “So you need to take all of your organization and match it with your application design. Your software creates a certain scale: it creates new clusters, it meets new use cases. But as developers, our current processes often fail to meet this new layer of speed. Our tools as developers haven’t really changed. So now we have reached a bottleneck, where if we try to increase our speed any faster, we get stuck by our capabilities. So we have to speed up our own monitoring processes and how we observe and iterate.”
Weis says that there are no must-have metrics today when monitoring in serverless, instead it needs to relate back to the application goals.
What are your use cases? Are they going to change over time? What are your goals? What are your next features? How will customers be using your product? What will you need in terms of observability, how much time will you have and how much time will you need? Those are the questions you need to ask before you start monitoring, said Weis. “It is no longer about asking about specific performance or the specific time that your function needs to run. Focusing on one metric will blind you from the full picture. That is what a lot of the DevOps is moving to: let’s remove the noise around the code.”
To help with this new approach, Weis and his team have built Rookout, an insertable monitoring tool that feeds logging data from serverless systems into existing data tools like Honeycomb, Datadog, and Sentry.io. While also working with bare metal microservices architectures, Rookout is increasingly focusing on AWS Lambda and IBM Cloud Functions for use in serverless systems.
The Rookout interface resembles an IDE, with four panes that show the source files, the code, a rules engine and the responses. Users can pull a source from a Git-based repository, and show the code in the main pane, they can then select a line where a new data collection log may be necessary. A range of defined rules for data collection have already been built, so devs don’t need to code up what data they want insight from with at that particular point in the code. The monitoring tool aims to work within the existing serverless ecosystem, so focuses on the collection and pipelining of data, allowing users to stream the data to their preferred tool, either the in-IDE response pane or their own data tools. Integrations exist for Honeycomb, feeding results directly to Slack and a range of other tools.
With Rookout, as well as identifying when the data log should run, users can also define what response they want to receive, including any specific parameters they need to see. “Basically, with AWS Lambda, this adds a dependency and then you wrap the function with our wrapper. It doesn’t slow down production in any way,” said Weis.
Because of the increasing uselessness of testing in development, the opportunity here is that the data logs can be returned and pushed to either a data tool or Slack, or another output interface while the functions continue to execute and run.
“That allows us to work with serverless in ways that weren’t possible before. You used to have to stop and see what was happening. Once you stop a function, your entire debugging system becomes useless, because it is no longer the actual processing environment. You get to view all the data that you want, but you get production grade quality data without interfering. It’s the infrastructure-as-code approach,” said Weis.
Feature image: Photo by Lou Levit on Unsplash.
