TNS
VOXPOP
Favorite Social Media Timesink
When you take a break from work, where are you going?
Instagram/Facebook
0%
Discord/Slack
0%
LinkedIn
0%
Video clips on TikTok/YouTube
0%
X, Bluesky, Mastodon et al...
0%
Web surfing
0%
I do not get distracted by petty amusements
0%
Data / Operations / Storage

DynamoDB vs. ScyllaDB: A Price Performance Comparison

To better assess whether moving off DynamoDB makes sense, we did a detailed comparison across workload conditions, distributions and read:write ratios
Dec 7th, 2023 8:30am by
Featued image for: DynamoDB vs. ScyllaDB: A Price Performance Comparison
Image via Pixabay.

Since early 2022, ScyllaDB has experienced a significant increase in the number of DynamoDB users moving to ScyllaDB Cloud, our database as a service offering. Price performance is cited as a primary driver in virtually all of these interactions. To help teams better assess whether a move makes sense, we decided to expand on our original DynamoDB benchmark with a detailed price-performance comparison analyzing:

  • How cost compares across both DynamoDB pricing models under various workload conditions, distributions and read/write ratios.
  • How latency compares across a variety of workload conditions.

You’ll find detailed findings below, but here’s the bottom line: ScyllaDB costs are significantly lower in all but one scenario. In realistic workloads, costs would be five to 40 times lower with up to four times better P99 latency.

Sample Workload Comparison: Uniform Distribution, 50:50 Read/Write Mix

Here is a consolidated look at how the DynamoDB and ScyllaDB compare on cost and performance for just one of the many workloads we tested. DynamoDB shines with a uniform distribution and struggles with the others; we chose to highlight a case where it shines.

Note: For ScyllaDB, we use on-demand pricing for both cost comparisons. “Provisioned throughput pricing” does not apply to ScyllaDB. Also, both vendors offer discounts for annual and multiyear commitments. ScyllaDB’s latencies were measured when utilization was at 75% capacity, where throughput was reported. Lower utilization results in better latencies.

Setup

We ran benchmarks using a slightly modified version of YCSB 0.18.0+ on ScyllaDB Enterprise 2022.2 vs. DynamoDB. Here’s what we provisioned:

Note that we opted to use relatively small deployments since we wanted to test throughput ranges to approximate average or smaller than average ScyllaDB deployments (100,000 operations per second). Previous extreme scale benchmarks demonstrated ScyllaDB’s ability to achieve single-digit millisecond latency with 7,500,000 operations per second.

The following loaders were used to run the Yahoo Cloud Serving Benchmark (YCSB) against both databases:

  • Number of loaders: Six to eight, depending on the test
  • Loader nodes type: c5.2xlarge
  • Region: us-east-1

Using YCSB, each database was preloaded with 1 billion rows, adding up to ~1 terabytes of data. The data model follows the standard YCSB “workload” definitions: 10 fields of data, named “field0” to “field9”, each of which has 100 bytes of information. This added up to a payload of approximately 1 KB per record.

We tested three workload distributions: uniform, Zipfian and hotspot distributions.

For complete details on the setup and configuration, see the extended ScyllaDB vs. DynamoDB report.

Cost Results

For our cost comparisons, we started with a range of throughput plus workload scenarios and calculated their cost on ScyllaDB Cloud with the AWS instances specified above. Next, we calculated the cost of running the same throughput plus workload scenarios on DynamoDB. We needed to know:

  • The item size to compute the write capacity units (WCUs) and read capacity units (RCUs) used per operation.
  • The monthly price for 1 terabyte of storage.
  • The cost per unit.

We used an item size of 1,081 bytes (calculated using the DynamoDB documentation), which translates to two WCUs per write operation and one RCU per read operation on DynamoDB. 1 TB costs around $250/month. The cost per unit varies according to the pricing model: provisioned or on demand.

For ScyllaDB we used a three-node cluster for all modes. Users can scale out/in the cluster at any point and are charged hourly rates (on-demand). As the platform scales with the amount of resources used, you can linearly change the price for the performance level you require. Annual pricing provides significant cost reduction but is out of the scope of this benchmark.

DynamoDB has two modes for non-annual pricing: provisioned and on-demand pricing.

Provisioned mode is recommended if your workloads are reasonably predictable. On-demand pricing is significantly more expensive and is a fit for unpredictable workloads. It is possible to combine modes, add auto-scaling and so forth. DynamoDB provides considerable flexibility around managing the cost and scale of the aforementioned options, but this also results in considerable complexity.

We measured the ScyllaDB performance in all different workload mixes and distributions, then compared it to the cost of DynamoDB with allocations to the required read/write units. Note that beyond ScyllaDB’s on-demand pricing, which can be estimated with our pricing calculator, discounts are provided for annual commitments. To see how we calculated costs, see the cost calculation details in the extended report.

ScyllaDB Utilization

The number of operations per second that the ScyllaDB cluster performs for each workload is reported under the X axis in the following graphs. Through all tests, we ensured that the ScyllaDB cluster utilization (load) was no more than 75% and reported its resulting latency. Given that, note that it is possible to achieve higher peak performance (up to two times the amount reported for shorter bursts) at no additional cost under a cluster of the same size, as well as achieve lower latencies with lower utilization.

Provisioned Mode Cost Comparisons

Provisioned mode is recommended if your workloads are reasonably predictable. With DynamoDB, you need to be able to predict per-table capacity regarding read and write capacity units.

With just one exception, DynamoDB’s cost estimates were consistently higher than ScyllaDB’s and much more so for the most write-heavy workloads. This is not surprising, given that DynamoDB charges five times more for writes than for reads, while ScyllaDB does not differentiate between operations and its pricing is based on the actual cluster size.

We also compared pricing using DynamoDB’s on-demand pricing model. Here, the same general trends held true. ScyllaDB cost was less across the board, and its cost advantage increased as the write percentage and throughput increased. For details, see the extended benchmark report.

Performance Results

Next, let’s shift the focus to performance. For use cases that require near real-time responses such as AdTech, messaging, gaming, IoT, P99 latency is critical for meeting service-level agreements (SLAs) and delivering an engaging user experience.

Uniform Distribution

Here are the performance results for the uniform distribution.

ScyllaDB’s latency was significantly lower than that of DynamoDB — at a fraction of the cost. An even larger reduction in P99 latency and mean latency (under 1 millisecond) could be achieved by using a larger ScyllaDB cluster and reducing the utilization from 75% to the 30%-50% range. Additionally, ScyllaDB comes with several options that might further improve latencies, such as the BYPASS CACHE extension, designed for workloads that don’t make effective use of caching, such as a workload that is much larger than the RAM with uniform distribution. Such options have not been applied for this performance testing.

Hotspot Distribution

Now, let’s shift to a more realistic distribution: hotspot. The hotspot distribution has some sets of values accessed more frequently than others. A hotspot distribution is an aggregation of two distributions, the Bernoulli distribution and one of two uniform distributions (“chosen” by the former distribution). Here, the records are partitioned into two sets: hot and cold. When generating an item from this distribution, first a “Bernoulli coin” is flipped to determine the next set to draw an item from, and then an item from this set is picked with uniform distribution.

As you can see, the discrepancy between DynamoDB and ScyllaDB latencies widened for these more realistic workloads. Since a portion of the requests had a higher chance of hitting an object from the hot set, the probability of ScyllaDB’s cache being touched during reads increased significantly. The results show that ScyllaDB P99 latencies decreased compared to the previous uniform results as the read ratio increased.

The next graph represents the P99 latency as a function of ScyllaDB’s utilization. Note how the P99 decreases along with a lower cluster load:

Zipfian Distribution

Finally, let’s look at the interesting case of Zipfian distribution. Zipfian distribution takes the hotspot to new levels by simulating access patterns with an exponential relation between items (a.k.a. “hotness”). As Alex Debrie, author of “The DynamoDB Book” notes, Zipfian is generally problematic for DynamoDB: “The most popular items are accessed orders of magnitude more than the average item. Because DynamoDB wants a more even distribution of your data, your application may get throttled as it tries to access popular items.”

As expected, DynamoDB performed poorly on these workloads.

As Debrie noted, the Zipfian distribution is likely to create hot partitions, an imbalance introduced by uneven item access from within the database. As hot partitions are a known antipattern, DynamoDB restricts the number of hits on the same partition (documented to be 3,000 RCUs and 1,000 WCUs per partition at the time of writing). Aware of this limitation and of Zipfian’s imbalances, we compared both ScyllaDB and DynamoDB at scale.

Once DynamoDB limits were reached, requests started getting throttled, requiring the application to retry. As a result of the throttling and retries, the YCSB latency became severely impacted. Under some circumstances, we experienced DynamoDB throttling at up to ~2.5 seconds per request, thus preventing the application from accessing the item during that time.

At best, DynamoDB delivered 39.72% of the expected throughput (88,190 good operations per second vs. the target 222,000 operations per second). At worst, it delivered only 16.22% (20,280 operations per second) of what was purchased (125,000 operations per second) and supposedly provisioned. Given the documented DynamoDB limitations, some discrepancy is to be expected. However, the magnitude of this unfulfilled throughput was surprising.

Unlike DynamoDB, ScyllaDB managed to sustain the target throughput without any throttling and still deliver single-digit millisecond latencies. In that regard, ScyllaDB does not impose any hard limits on querying hot partitions. Even though ScyllaDB implements concurrency-limiting mechanisms, frequently accessing popular items will typically benefit from its cache implementation, thus explaining why no failures have been seen during the Zipfian tests. Even then, it is worth underscoring that hot partitions are an antipattern, even for databases like ScyllaDB. Read more about ScyllaDB’s advanced control mechanisms in ScyllaDB in this blog.

Advanced Configurations

Caching: DynamoDB Accelerator (DAX)

Given the latency results, it is interesting to note that teams using DynamoDB with latency-sensitive workloads and strict P99 requirements might require relying on DynamoDB Accelerator (DAX) to achieve their SLA targets. However, caching solutions add a significant increase to DynamoDB costs and complexity, driven by the constant need to monitor cost, reserved capacity and scaling. ScyllaDB does not require any external caching component to sustain predictable low latencies. As mentioned earlier, ScyllaDB’s BYPASS CACHE feature allows the user to give the database hints about which query results should not be cached, thus protecting against cache invalidation.

Autoscaling: A Mix of Provisioned and On-Demand

Once the DynamoDB cost grows, users often start to optimize it with a set of tools that AWS offers: combining on-demand and provisioned workloads, auto-scaling, etc. However, this not only increases complexity, it could also incur limitations from the speed of elasticity and even throttling. Any cost savings achieved by adding this complexity are highly unlikely to close the large cost gap between the two databases.

Making ScyllaDB Even More Cost-Effective

Unlike DynamoDB, where you provision tables, ScyllaDB is provisioned as a cluster, capable of hosting several tables and therefore consolidating several workloads under a single deployment. Excess hardware capacity may be shared to power more use cases on that cluster. Users may rely on ScyllaDB’s Workload Prioritization feature to assign different priorities to different tables or to different use cases that share a single table.

For example, assume there are 10 use cases that require 100,000 operations per second each. However, at any given point in time, only two of them will reach their limit. With DynamoDB, users would be forced to allocate a provisioned workload per table or to use the rather expensive on-demand mode. A standard ScyllaDB deployment is not only more cost-effective, it also allows users to run all 10 workloads within a single cluster with a sustained capacity of 200,000-300,000 operations per second and to share the idle time among all those workloads. Prioritized workloads will receive more CPU and IO resources from the scheduler, resulting in higher throughput and lower latencies.

Conclusion

As the results indicate, what might begin at a seemingly reasonable cost can quickly escalate into “bill shock” with DynamoDB, especially as the throughput increases and particularly with write-heavy workloads. This makes it a suboptimal choice for data-intensive applications anticipating steady or rapid growth. ScyllaDB’s significantly lower costs, a reflection of ScyllaDB taking full advantage of modern infrastructure for high throughput and low latency, make it a more cost-effective solution for data-intensive applications.

ScyllaDB — with its log-structured merge-tree (LSM) based storage, unified caching, shard-per-core design and advanced schedulers — allows you to maximize the advantages of modern hardware, from huge CPU chips to blazing-fast nonvolatile memory express (NVMe).

This small-scale benchmark demonstrated how a 57,000 operations per second workload with a 50:50 read/write ratio that costs $4,500/month on-demand on ScyllaDB would cost $30,000 to $200,000/month with DynamoDB. Beyond those cost savings, ScyllaDB sustains two-times peaks and provides two to four times better P99 latency. Additionally, it can further reduce latency when idle or enable spare resources to be shared across multiple tables. For larger workloads spanning 500,000 to 1,000,000 operations per second and beyond, this can result in a cost saving in the millions with better performance and fewer query limitations.

Additional Details about the Benchmark

See the extended DynamoDB vs ScyllaDB benchmark report for additional details on:

  • Methodology (including considerations, setup and workloads).
  • Our YCSB configuration and modifications.
  • How we calculated costs across both provisioned and on-demand pricing models.
  • A deeper dive into distributions (uniform, hotspot and Zipfian).
Group Created with Sketch.
TNS owner Insight Partners is an investor in: Uniform.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.