found out my LLM feature's cost problem from an invoice, not a dashboard, and that felt very wrong
by camerongreen95
shipped an LLM feature, felt fine at launch, moved on to other work. a while later the bill showed up noticeably higher than expected and that was the first real signal anything had changed. no alert, no dashboard flag, just a number at the end of the month that made me go "wait, what happened." went back and actually set up proper tracing and token/cost monitoring per request instead of just trusting it'd be fine, and found a specific workflow was making way more calls than i thought due to a retry loop that wasn't being logged anywhere visible. it had been quietly running up cost for weeks with zero visibility until the invoice. also added latency budgets and caching for repeated queries after this, which cut cost noticeably on top of the retry fix. feels like a pretty basic devops instinct (you monitor what you ship, you don't wait for the bill) that somehow gets skipped constantly once "AI feature" is involved, like people ship LLM stuff with less observability discipline than they'd ever accept for a normal service. anyone else's team caught something similar the hard way before actually building proper monitoring in from the start submitted by /u/camerongreen95 [link] [comments]