←back to thread

How We Found 7 TiB of Memory Just Sitting Around

(render.com)
205 points anurag | 2 comments | 30 Oct 25 18:25 UTC | HN request time: 0s | source
Show context
Aeolun ◴[01 Nov 25 02:54 UTC] No.45778883[source]▶
I read this and I have to wonder, did anyone ever think it was reasonable that a cluster that apparently needed only 120gb of memory was consuming 1.2TB just for logging (or whatever vector does)
replies(4): >>45779398 #>>45779624 #>>45779726 #>>45780688 #
bstack ◴[01 Nov 25 05:08 UTC] No.45779398[source]▶
Author here: You’d be surprised what you don’t notice given enough nodes and slow enough resource growth over time! Out of the total resource usage in these clusters even at the high water mark for this daemonset it was still a small overall portion of the total.
replies(2): >>45779736 #>>45779879 #
Aeolun ◴[01 Nov 25 07:39 UTC] No.45779879[source]▶
I’m not sure if that makes it better or worse.
replies(2): >>45780858 #>>45785752 #
1. embedding-shape ◴[01 Nov 25 11:30 UTC] No.45780858{3}[source]▶
I didn't know what Render was when I skimmed the article at first, but after reading these comments, I had to check out what they do.

And they're a "Cloud Application Platform" meaning they manage deploys and infrastructure for other people. Their website says "Click, click, done." which is cool and quick and all, but to me it's kind of crazy an organization that should be really engineering focused and mature, doesn't immediately notice 1.2TB being used and tries to figure out why, when 120GB ended up being sufficient.

It gives much more of a "We're a startup, we're learning as we're running" vibe which again, cool and all, but hardly what people should use for hosting their own stuff on.

replies(1): >>45791424 #
2. Anonbrit ◴[02 Nov 25 16:25 UTC] No.45791424[source]▶
If your report for the month is "I saved a terabyte of ram usage across our cluster estate!" and I as a manager do some quick maths and say great, that's our income from 2 median customers. We lost 8 customers because we didn't laugh feature foo in time, which is what you were supposed to be working on, so your contribution for the month is a massive loss to the company...

Does that frame things differently? There's are times in your product lifecycle where you doing want your developers looking at things like this, and a time when you do