WebSockets cost us $1M on our AWS bill

Classic story of a startup taking a "good enough" shortcut and then coming back later to optimize.

---

I have a similar story: Where I work, we had a cluster of VMs that were always high CPU and a bit of a problem. We had a lot of fire drills where we'd have to bump up the size of the cluster, abort in-progress operations, or some combination of both.

Because this cluster of VMs was doing batch processing that the founder believed should be CPU intense, everyone just assumed that increasing load came with increasing customer size; and that this was just an annoyance that we could get to after we made one more feature.

But, at one point the bean counters pointed out that we spent disproportionately more on cloud than a normal business did. After one round of combining different VM clusters (that really didn't need to be separate servers), I decided that I could take some time to hook up this very CPU intense cluster up to a profiler.

I thought I was going to be in for a 1-2 week project and would follow a few worms. Instead, the CPU load was because we were constantly loading an entire table, that we never deleted from, into the application's process. The table had transient data that should only last a few hours at most.

I quickly deleted almost a decade's worth of obsolete data from the table. After about 15 minutes, CPU usage for this cluster dropped to almost nothing. The next day we made the VM cluster a fraction of its size, and in the next release, we got rid of the cluster and merged the functionality into another cluster.

I also made a pull request that introduced a simple filter to the query to only load 3 days of data; and then introduced a background operation to clean out the table periodically.

As much as you can say (perhaps not hard numbers, but as a percentage), what was the savings to the bottom line / cloud costs?

Probably ~5% of cloud costs. Combined with the prior round of optimizations, it was substantial.

I was really disappointed when my wife couldn't get the night off from work when the company took everyone out to a fancy steak house.

99% of the time, it's either a quadratic (or exponential) algorithm or a really bad DB query.

So you saved the company $10k a month and got a $200 meal in gratitude? Awesome.

can also be a linear algorithm that does N+1 queries. ORMs can be very good at hiding this implementation detail

What should they have gotten?

They're presumably already being paid a salary to do this work.

They're more pissed about the 1.2M they spent than about the 10k a month they saved.

> So you saved the company $10k a month and got a $200 meal in gratitude? Awesome.

You seem to be assuming that a $200 meal was the only compensation the person received, and they weren't just getting a nice meal as a little something extra on top of getting paid for doing their job competently and efficiently.

But that's the kind of deal I make when I take a job: I do the work (pretty well most of the time), and I get paid. If I stop doing the work, I stop getting paid. If they stop paying, I stop doing the work. (And bonus, literally, if I get a perk once in a while like a free steak dinner that I wasn't expecting)

It doesn't have to be more complicated than that.

They are in theory owed nothing more than their salary but it can be very good for moral to reward that type of thing (assuming they are not introducing a perverse incentive)

Yeah? Well, proper rewards make those savings and optimizations more common. Otherwise most people will do the work needed just to have work tomorrow.

> a linear algorithm that does N+1 queries.

That's what quadratic means.

It creates a perverse incentive to deliberately do things a more expensive way at the beginning and then be a hero 6 months down the line by refactoring it to be less expensive.

Typically implemented accidentally: https://www.tumblr.com/accidentallyquadratic

Depends. There are people who put in the absolute minimum work they can get away with, and there are people who have pride in their profession.

That's independent of pay scale.

Granted, if you pay way below expectations, you'll lose the professionals over time. But if you pay lavishly no matter what, you get the 2021/2022 big tech hiring cycle instead. Neither one is a great outcome.

I'm not sure how they feel, but when it happens to me, it's not a big deal because it's my job to do things like that. If I fuck up and cost them $10k/month I'm certainly not going to offer to reimburse them.

Not really what they're talking about https://stackoverflow.com/questions/97197/what-is-the-n1-sel...

Ha ha, software developers already have this incentive. Viz: "superhero 10x programmer" writing unmaintainable code to provide some desirable features, whose work later turns out to be much less awesome than originally billed.

Of course the truth is more complicated than the sound bite, but still...

Great.

It will be great if anyone write a checklist(playbook) to be checked for CPU, Memory, Disk, IO and network issues.