Limiting the Cost of Scale
This is part of a series of contributed articles leading up to KubeCon + CloudNativeCon on Oct. 24-28.
Over the past 10 years, we’ve seen the popularity of Kubernetes and microservices boom, generating more data that is often never used. The amount of observability data has doubled, and in some cases even tripled, as the technologies that we use become more complex.
Much of this data is stored for “when it might become useful next month.” However, most of this data will not become useful next month or even next year. The cost of storing this unused data is huge. But is the cost of not having this data larger? To allow businesses to store as much data as they need, without crippling costs, it’s crucial we limit the cost of scale.
Where to Store All That Extra Data
When we talk about the average number of metrics continuously growing, we’re talking about data such as server metrics, application performance monitoring, network data, sensor data, events, clicks and many other types of analytics data. This is where time series databases (TSDB) come in, as they are optimized for timestamped or time series data. Time series data are simply measurements or events that are tracked, monitored, downsampled and aggregated over time.
Prepping for Scale
When looking for a highly scalable time series database, there are a number of criteria you should investigate and evaluate. Choosing open source software should be an obvious starting point. It’s more likely to have gone through comprehensive troubleshooting, is typically more reliable, better guarantees technology independence, and it has great security. Not to mention it’s easier to find engineers who are familiar with it.
Once you’ve settled on using an open source time series database, you must identify your key criteria for scaling. For example, is multitenancy a key driver? Or perhaps ensuring high availability or cost-effectiveness? Beyond these factors, what functionalities do you need — thorough and honest documentation? Downsampling of historical data for more efficient storage? These are essential questions you must be able to answer before you investigate which solution you should use.
Choosing the Most Scalable Time Series Database
Once you’ve identified your business needs, it’s important to do your due diligence and fully test potential solutions for key indicators that could affect the scaling efficiency. For example, are there bottlenecks in query or ingestion performance? How does the solution deliver efficient compression? How does the resource usage change as you scale? How does the scaling factor affect resources?
Focusing on scaling factors — the best characteristic to look out for is linear scaling which means if you double resources then the capability of the system is also doubled. Unfortunately, this is rare in distributed systems. Most systems have a coefficient of the scaling, which means if you double resources you get only 30% of additional capability. Resource usage can be a determining factor both from a system as well as from a budgetary perspective. Great scale can quickly lead to great cost, but it doesn’t have to.
Object Storage vs. Local File System
Object storage in the cloud is the new approach whereas storing via a local file system is the traditional approach. Object storage only recently became an option with the rise of AWS, which can make it quite cheap. But there are, of course, pros and cons to each.
Object storage is infinite, durable, and (generally) cheap for storing data — you pay only for what you use. However, it’s much slower than local storage. It has significantly higher latency than a local file system, which means the network bandwidth could become a bottleneck. There also can be significant data transfer and data retrieval costs.
Unlike object storage, a local file system has none of these hidden costs. It is the best option for really good read performance delivering super-fast — almost instant — backups, which is only possible on local file systems.
A local file system also allows for mutating (merging) of data aggressively to achieve better compression on disk. But there are some disadvantages: It has a higher price per gigabyte with limited size that requires resizing when needed, and it also requires additional durability guarantees. However, many of the cloud providers such as Google, AWS and Azure do provide additional availability guarantees when using their disks.
Understanding the drivers behind your need to scale and the indicators you’re looking for will help you decide which of the above is the best route to go down. However, for most, the local file system setup provides the best options for outstanding read and write performance.
Betting on a Simple Scalable Future
The key to cost-efficient scalability is simplicity and transparency. The less magic happens under the hood and the fewer components used, the better the efficiency. It’s not only about efficient usage of hardware resources, but also about the amount of effort engineers need to understand and maintain the software.
Monitoring should be as accessible as possible to everyone who needs it. And yet many of the solutions on the market are complex to handle with moving components and extra dependencies. Performance, simplicity and reliability in delivering both high scalability and availability should be front of mind for any business looking for a time series database solution.
No matter which open source time series database solution you choose, its documentation should contain clear tips on things like scaling cluster components or recommendations and performance expectations based on the amount of resources provided and workload volume. A good solution should constantly improve its systems and processes in response to customer feedback and performance reports.
Understanding the important performance indicators that meet your business’ data scaling requirements will help you find the best solution. Blindly picking a solution based on other people’s needs will not.
To hear more about cloud native topics, join the Cloud Native Computing Foundation and the cloud native community at KubeCon + CloudNativeCon North America 2022 in Detroit (and virtual) from Oct. 24-28.
