←back to thread

Datadog's $65M/year customer mystery solved

(blog.pragmaticengineer.com)
151 points thunderbong | 1 comments | | HN request time: 0.204s | source
Show context
ljm ◴[] No.44427444[source]
I wonder how much that no-expense-spared, money-is-no-object attitude to buying SaaS impacts an engineers ability to make sensible decisions around infra and architecture. Coinbase might have been fine blowing 65 mil but take that approach to a new startup and you could trivially eat up a significant amount of runway with it.

I won’t single out Datadog on this because the exact same thing happens with cloud spend, and it’s very literally burning money.

replies(4): >>44427650 #>>44428240 #>>44428533 #>>44428683 #
viccis ◴[] No.44428240[source]
>I wonder how much that no-expense-spared, money-is-no-object attitude to buying SaaS impacts an engineers ability to make sensible decisions around infra and architecture

I saw this a lot at a previous company. Being able to just "have more Lambdas scale up to handle it" got some very mediocre engineers past challenges they encountered. But it did so at the cost of wasting VAST amounts of money and saddling themselves with tech debt that completely hobbled the company's ability to scale.

It was very frustrating to be too junior to be able to change minds. Even basic things like "I know it worked for you with old on-prem NFS designs but we shouldn't be storing our data in 100kb files in S3 and firing off thousands of Lambda invocations to process workloads, we should be storing it in 100mb files and using industry leading ETL frameworks on it". They were old school guys who hadn't adjusted to best practices for object storage and modern large scale data loads (this was a 1M event per second system) and so the company never really succeeded despite thousands of customers and loads of revenue.

I consider cost consideration and profiling to be an essential skill that any engineer working in cloud style environments should have, but it's especially important that a staff engineer or person in a similar position have this skill set and be ready to grill people who come up with wasteful solutions.

replies(2): >>44432069 #>>44432131 #
nasmorn ◴[] No.44432069[source]
It is also not a very hard skill. You do a back of the envelope calculation and if your proposed architecture is crazy expensive for your reasonable load, then you have to figure out if you are a special snowflake or just doing it wrong.
replies(1): >>44438048 #
1. viccis ◴[] No.44438048[source]
This is correct. It's really more of a mindset than anything. You take a guess at how much something will cost based on a quick calculation (good cloud providers make this easy, some cough Databricks cough just use a black box and bill you whatever they feel like) and then once you test it at a small scale, you verify that it's as expected and continue to monitor.