What news from AWS re:Invent last week will have the most impact on you?
Amazon Q, an AI chatbot for explaining how AWS works.
Super-fast S3 Express storage.
New Graviton 4 processor instances.
Emily Freeman leaving AWS.
I don't use AWS, so none of this will affect me.
Cloud Services / Observability / Operations / Software Development

Datadog’s $65M Bill and Why Developers Should Care

Observability bills can surprise companies, and developers may be on the hook for changing code if bills grow too large.
May 17th, 2023 10:56am by
Featued image for: Datadog’s $65M Bill and Why Developers Should Care
Photo by Andrea Piacquadio via Pexels

Sixty-five million dollars: That’s how much one customer was billed by Datadog in the first quarter of 2022, according to a May 4 earnings call for the SaaS observability and security vendor.

An unspecified financial services firm was faced with the bill, which did not reoccur, according to Olivier Pomel, co-founder, CEO and Director at Datadog.

“We had a large upfront bill for a client in Q1 ’22 that did not recur at the same level or timing in Q1 ’23,” he said, during the earnings call. “Pro forma for this client, billings growth was in the low 30s percent year-over-year.”

Pro forma refers to a financial statement that uses hypothetical data or assumptions about future values to project performance over a period that hasn’t yet occurred, according to the Harvard Business School. Generally, observability companies use this approach to predict what a bill may be — but as the $65 million bill shows, there can be surprises when the actual usage bill comes due.

Mark Ronald Murphy, an executive director and financial analyst with JPMorgan Chase’s research division, crunched the numbers and revealed that the upfront bill would have been about $65 million.

David Obstler, chief financial officer at Datadog, said the company changed the billing frequency.

“That customer’s bill will, one, be spread out more over time. That company — that was a crypto company and continues to be a customer of ours, but that was an early optimizer,” Obstler said, during the earnings call. “We will get that bill at a smaller size than was billed last year in a more of a chunked up billing way.”

He added that since becoming public, Datadog has pointed out when they have an unusual bill and that the problem isn’t common. When it does happen, they change the duration or timing or size of the bill to accommodate customers.

Was Coinbase the Customer?

Pomel added that the customer was in a vertical that was “pretty much decimated over the past year.”

“Their own business was cut in three or four in terms of their revenue and when that’s the case, that we really work with customers to restructure their contract with us,” Pomel said. “We want to be part of the solution for them, not part of the problem. And that’s what we did here, we restructured that contract. So we kept them as a happy customer for many more years and do deal that works for everyone with their business profile.”

While Datadog did not return an interview request by deadline, Gergely Orosz, a software developer who blogs as the Pragmatic Engineer, cites multiple unnamed engineering sources at Coinbase whom he said confirmed Coinbase was the company in question. Coinbase did not respond to The New Stack’s request to confirm or deny whether they were the company in question.

Observability Costs: The Impact on Developers

In Datadog’s case, the numbers are complicated by the fact the company offers more than observability solutions, including security bills. The earnings report did not clarify how many such SaaS services the unnamed company used.

While $65 million is a shocking amount — shocking enough that the news quickly circulated on Twitter — bills in the ten million range are not unusual for traditional observability companies, said Shahar Azulay, CEO of observability alternative provider Groundcover.

“Big companies, like Coinbase, have already proceeded to the $10 million per year, price tag a while ago,” Azulay said. “It’s not rare to hear companies paying Splunk, Dynatrace, Datadog — all like the big observability players — paying them over $10 million a year and even paying multiple vendors, each of them above like two figures per year.”

Part of it is how observability companies choose to price their offerings, he added. Observability solutions monitor three types of data: Logs, metrics and traces (which monitor the pathways for interactions, such as end-to-end transactions and what happens between services). It’s difficult if not impossible to predict how these data sources will grow, particularly when they might be spiked by events such as Black Friday, when customer usage peaks.

“It has tons of unpredictability and a lot of dependency on the amount of data you push to your log, and that’s the base root cause on these huge volumes of pricing points because you’re not being able to control that and you’re not being able to know how much you’re going to pay next month,” Azulay said.

What’s more, even if a contract is for one tier level, once a company exceeds that tier, it’s billed at the higher tier rate from that day forward, he added.

“That specific logline can be a critical part of the infrastructure, say, a search engine in Google or whatever that runs a million times a day — just customers using it a million times a day,” he said. “Suddenly, from an organization perspective, you could have just pushed like a million more log lines or data points into data without knowing that as a developer. It creates a cycle of developers creating applications, building business logic that supports what the organization should be doing as a product, and then R&D management, figuring out two months later, ‘Oh, that just spiked our prices by 50%.”

That may fall back on the developer for pushing too much information to the observability stack, he said.

“They’re causing the developers to cut down on the number of data points they push to monitor the production,” Azulay said. “It’s a weird, vicious cycle of developers wanting more data to troubleshoot, and management being put in a trade-off where they have to pay tons of money for that.”

Not all observability companies charge this way. Groundcover, which uses an eBPF Agent for observability, collects the data but stores only what matters, which it says is more cost effective, so it can charge by the number of servers running in production, Azulay said.

More Datadog Deals

Earnings reports provide an opportunity to learn about the inner workings of public companies. For instance, the Datadog first-quarter earnings report shows that revenue during that quarter was $482 million — an increase of 33% year over year despite a March service outage that reduced the quarter’s revenue by $5 million. That outage required three shifts of 500-600 engineers working on the outage, Pomel said. Billings were $511 million for the first quarter up 15% year over year, Pomel said.

The company ended the quarter with 25,500 customers, up 19,800 over the same quarter last year. Of those, about 2,910 have an annual reoccurring revenue (ARR) of $100,000 or more, up from about 2,250 last year. These customers generate about 85% of Datadog’s annual reoccurring revenue. The majority of customers — 81% — also use two or more Datadog products.

In the first quarter of the year, Datadog executives reported signing an eight-figure deal with a leading AI company; a seven-figure expansion with a Fortune 500 health care company; a seven-figure multiyear deal with a leading university in Australia that had historically relied on open source solutions; and an expansion deal to another eight-figure ARR deal with “one of the world’s largest fintech companies.

“This customer has expanded meaningfully over time, and today see Datadog platform used by thousands of users across dozens of business units,” Pomel reported. “With this expansion, this customer now uses 14 Datadog products and is consolidating multiple open source, homegrown and commercial tools across observability and security into the Datadog platform.”

It also provided a peek at why companies might be willing to shell out eight-figure fees to the company. Before Datadog, Pomel said the Fortune 500 health care company would need to mobilize upper to 150 employees for an average of 3-4 hours for the same function that now requires only 20 employees for about 30 minutes.

Correction: The story has been updated to reflect the correct capitalization of the second d in Datadog; also Groundcover does collect data, but doesn’t store all data.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: The New Stack.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.