LLM Observability in the Wild – Why OpenTelemetry Should Be the Standard

LLM app telemetry is important, but I don’t think we have seen the right metrics yet. Nothing has convinced me that they are more useful than modern app telemetry

I don’t think tool calls or prompts or rag hits are it

That’s like saying that C++ app observability is about looking at every sys call and their arguments

Sure, if you are the OS it’s easy to instrument that, but IMO I’d rather just attach to my app and look at the logs

Attaching to the app is impractical to catch regressions in production. LLMs are probabilistic - this means you can have a regression without even changing the code / making a new deployment.

A metric to alert on could be task-completion rate using LLM as a judge or synthetic tests which are run on a schedule. Then the other metrics you mentioned are useful for debugging the problem.