←back to thread

116 points ndhandala | 1 comments | | HN request time: 0.203s | source
Show context
N_Lens ◴[] No.45082128[source]
OTEL as a set of standards is admirable and ambitious, though in my experience actual implementation differs significantly between different vendors and they all seem to overcomplicate it.
replies(1): >>45082689 #
eurekin ◴[] No.45082689[source]
Plus that tens of terabytes of data you have to store for a week worth of traces
replies(1): >>45083511 #
c2h5oh ◴[] No.45083511[source]
That's why you sample just enough instead of storing everything
replies(3): >>45083636 #>>45083681 #>>45083814 #
voidfunc ◴[] No.45083636[source]
That sounds great until you have a massive issue that costs the company real money and leadership asks why you weren't logging everything in full fidelity?

We run with Debug logging on in prod for that reason too. We also ingest insane amounts of data but it does seem to be worth it for a sufficiently complex and important enough system to really have it all.

replies(3): >>45084058 #>>45084941 #>>45094434 #
evidencetamper ◴[] No.45084058[source]
> and leadership asks why you weren't logging everything in full fidelity?

I haven't been asked this question ever. In a way, I wish I was. I wish leadership was engaged in the details of the capabilities of the systems they lead.

But I don't anyone asking me this question any time soon either.

replies(1): >>45085449 #
no_wizard ◴[] No.45085449[source]
Have you ever been asked “why didn’t we catch this sooner?”. I feel like it’s the same question worded differently
replies(1): >>45089399 #
1. voidfunc ◴[] No.45089399[source]
Its really two questions:

1. Why didn't we catch this sooner

2. Why did it take so long to mitigate

Without the debug logging #2 can be really tricky sometimes as well as you can be flying blind to some deep internal conditional branch firing off.