CommPulse

The Qdrant snapshot API returns 200 OK with a JSON body containing a filename, a size, and a creation timestamp. That response confirms a tar archive got written somewhere inside the container. It says nothing about whether that somewhere survives a pod restart, and with the manifest most people start from, it doesn't. That's the whole problem in one sentence. The API is honest about what it did. It has no opinion about where your storage lives, and neither does your monitoring, which is why a broken Qdrant backup can report success for a very long time before anyone finds out. Who should care If you're running Qdrant in Kubernetes as the vector store behind a RAG pipeline, an agent memory layer, or a semantic search index, the embeddings in it are expensive. Not expensive to store, expensive to regenerate . Re-embedding a few hundred thousand chunks means re-chunking the source documents, re-running the embedding model, and hoping the model version you used originally is still pinned somewhere. A restore from a 20 MB snapshot file takes ninety seconds. A rebuild takes an afternoon and a stack of GPU hours. This post walks the progression from the naive setup to one that actually holds: emptyDir → PersistentVolumeClaim → external NFS target, plus the retention logic that keeps the whole thing from eating your cluster. Stage 1: the emptyDir trap Here's the deployment shape you'll find in a dozen blog posts and at least two Helm chart examples: volumes : - name : qdrant-storage emptyDir : {} containers : - name : qdrant image : qdrant/qdrant:latest volumeMounts : - name : qdrant-storage mountPath : /qdrant/storage Nothing about that manifest is wrong for a demo. It's wrong for anything you care about, and the failure mode is worse than "I lose my data on restart" because of where Qdrant puts snapshots by default. Qdrant writes snapshots to ./snapshots relative to its working directory, which in the official image resolves to /qdrant/snapshots . That path is not /qdrant/storage . So even if you got the storage volume right and skipped emptyDir for the data, an unmounted snapshots path means your snapshots land on the container's writable layer, which is destroyed on every pod recreation. Node drains during a cluster upgrade, an OOM kill, a rollout from a changed config map, all of it wipes the snapshot directory silently. The check takes one command: kubectl exec -n vectordb deploy/qdrant -- df -h /qdrant/storage /qdrant/snapshots If either path shows overlay as the filesystem, that directory is ephemeral. If /qdrant/snapshots shows the same device as /qdrant/storage , you have a different problem, which is stage 2. Stage 2: the PVC that isn't a backup The obvious fix is to put both paths on a PersistentVolumeClaim. Data survives restarts, snapshots survive restarts, everyone goes home. Except now your backups live on the same block device as the thing they're backing up. I wrote about this failure shape in Your Vector DB Snapshots Are Landing on the Same Disk That Will Fail , and it's the single most common mistake in self-hosted vector database setups. A snapshot on the same volume protects you against exactly one class of event: application-level corruption. Someone drops a collection, an ingestion job writes garbage vectors, a schema migration goes sideways. Fine, restore from the local snapshot. It protects you against nothing else. Replica corruption, a Longhorn volume that won't attach, a filesystem that mounts read-only after an unclean shutdown, an accidental kubectl delete pvc with a Delete reclaim policy. In every one of those cases the snapshot goes down with the ship. There's a second, sneakier problem with co-locating snapshots on the data PVC: capacity. Qdrant snapshots are not incremental and they are not deduplicated. Creating a snapshot of a collection requires roughly as much free space as the collection itself, because Qdrant is tarring up the segment files. A 20 GiB PVC holding 12 GiB of vectors cannot take a snapshot. You get a failed request and a partially written archive that still occupies space until something cleans it up. Nothing cleans it up. Qdrant has no built-in retention for collection snapshots. Every call to POST /collections/{name}/snapshots creates a new file with a fresh timestamp and leaves every previous one exactly where it was. Wire that to an hourly CronJob, forget about it for a few weeks, and you end up with north of a hundred snapshot archives on a volume sized for the live data. The PVC fills, Qdrant's write path fails, and the failure looks like an application bug rather than a storage one. Stage 3: snapshots leave the cluster's failure domain The pattern that holds separates three things that people tend to collapse into one: Layer Lives on Protects against Qdrant data Longhorn RWO PVC nothing, this is the live copy Qdrant snapshot (transient) same PVC, deleted after export app-level corruption, fast rollback Exported snapshot external NFS share disk failure, volume loss, cluster loss The transient snapshot is the important nuance. You still create the snapshot locally, because that's the only interface Qdrant gives you, but you treat the local copy as a temporary artifact with a lifetime of a few minutes rather than as the backup. Do not mount NFS as Qdrant's storage path Before the manifests, the caveat that will save you a week: Qdrant's storage directory needs a local block device. Qdrant memory-maps segment files and uses RocksDB for payload storage, and neither of those behaves well over NFS. File locking semantics differ, mmap over NFS has a very different consistency model, and the performance drop on HNSW search is severe enough that you'll notice it in p99 latency immediately. Qdrant's own documentation recommends against network storage for the data path. NFS is fine as a destination for finished snapshot archives. Those are sequential writes of an immutable tar file, which is the workload NFS is actually good at. The data volume apiVersion : v1 kind : PersistentVolumeClaim metadata : name : qdrant-data namespace : vectordb spec : accessModes : [ ReadWriteOnce ] storageClassName : longhorn resources : requests : storage : 40Gi # 2x expected collection size, snapshots need headroom That sizing rule matters. If your collections total 15 GiB, a 20 GiB PVC will fail to snapshot. Size for double, or move the snapshots path to its own smaller volume. The NFS target An NFS-backed PV, statically provisioned, mounted only by the backup job: apiVersion : v1 kind : PersistentVolume metadata : name : qdrant-backups-nfs spec : capacity : storage : 200Gi accessModes : [ ReadWriteMany ] persistentVolumeReclaimPolicy : Retain # never let K8s delete the backups nfs : server : 10.0.0.50 path : /export/backups/qdrant mountOptions : - nfsvers=4.1 - hard # block on server outage instead of returning EIO mid-write - timeo=600 - retrans=2 hard over soft is deliberate. A soft mount returns an I/O error after the timeout, which means a partially written snapshot archive that looks like a file and restores like a coaster. A hard mount blocks, the CronJob hits its activeDeadlineSeconds , and you get a clean failure you can alert on. Retain on the reclaim policy is the other non-negotiable. If someone deletes the PVC, you want the PV to go to Released and the data on the NFS server to stay exactly where it is. The export job The CronJob runs in its own pod and never touches Qdrant's PVC. That's not just hygiene, it's a hard constraint: a Longhorn RWO volume can only be attached to one node at a time, so a backup pod scheduled somewhere else physically cannot mount it. Everything moves over the HTTP API instead. apiVersion : batch/v1 kind : CronJob metadata : name : qdrant-snapshot-export namespace : vectordb spec : schedule : " 0 */6 * * *" concurrencyPolicy : Forbid failedJobsHistoryLimit : 5 jobTemplate : spec : activeDeadlineSeconds : 3600 backoffLimit : 2 template : spec : restartPolicy : OnFailure containers : - name : export image : curlimages/curl:8.11.0 command : [ " /bin/sh" , " /scripts/export.sh" ] env : - name : QDRANT_URL value : " http://qdrant.vectordb.svc.cluster.local:6333" - name : QDRANT_API_KEY valueFrom : secretKeyRef : { name : qdrant-api , key : api-key } volumeMounts : - { name : backups , mountPath : /backups } - { name : scripts , mountPath : /scripts } volumes : - name : backups persistentVolumeClaim : { claimName : qdrant-backups } - name : scripts configMap : { name : qdrant-export-script } concurrencyPolicy: Forbid prevents a slow export from stacking on top of the next scheduled run, which is how you accidentally create three concurrent snapshots and fill the data PVC. The script itself does four things: create, download, verify, delete the local copy. #!/bin/sh set -eu AUTH = "api-key: ${ QDRANT_API_KEY } " DEST = "/backups/ $( date -u +%Y-%m-%d ) " mkdir -p " $DEST " COLLECTIONS = $( curl -sf -H " $AUTH " " $QDRANT_URL /collections" \ | grep -o '"name":"[^"]*"' | cut -d '"' -f4 ) for C in $COLLECTIONS ; do # wait=true blocks until the archive is fully written SNAP = $( curl -sf -X POST -H " $AUTH " \ " $QDRANT_URL /collections/ $C /snapshots?wait=true" \ | grep -o '"name":"[^"]*"' | cut -d '"' -f4 ) curl -sf -H " $AUTH " \ " $QDRANT_URL /collections/ $C /snapshots/ $SNAP " -o " $DEST / $SNAP " # a truncated tar is worse than no backup, so check before pruning if ! tar -tf " $DEST / $SNAP " > /dev/null 2>&1 ; then echo "corrupt archive for $C , keeping remote snapshot" > &2 exit 1 fi sha256sum " $DEST / $SNAP " >> " $DEST /SHA256SUMS" curl -sf -X DELETE -H " $AUTH " \ " $QDRANT_URL /collections/ $C /snapshots/ $SNAP " done The tar -tf check is the part people skip. A snapshot that downloaded halfway because the NFS server hiccuped produces a file with a plausible name and a plausible-ish size, and you will not find out it's unreadable until the day you need it. Validating the archive before deleting the source means a failed export leaves the local snapshot in place and the job retries. Retention, on the NFS side Qdrant won't prune for you and neither will NFS. A second, tiny CronJob handles it: # keep 14 days of dailies, plus the first snapshot of each month forever find /backups -maxdepth 1 -type d -mtime +14 \ ! -name " $( date -u +%Y-%m ) -01" -exec rm -rf {} + Six-hourly snapshots at 14 days retention on a 3 GiB collection set works out to about 170 GiB before compression, which is why the PV above is sized at 200 GiB. Run the arithmetic for your own collection sizes before you pick a schedule, because the number gets ugly fast with hourly snapshots. Why the separation actually matters The word to keep in your head is blast radius . Every backup design is really a claim about which failures it survives, and the honest way to evaluate one is to name the failure and trace where the bytes are. A Longhorn recurring snapshot job is not a substitute here, and the reason is worth understanding. Longhorn snapshots are block-level, they live inside the same replicas as the volume data, and they form a chain. They're excellent for fast rollback and they cost almost nothing to create. They are not off-box copies. If a replica's underlying disk dies, the snapshot chain on that disk dies with it. Longhorn backups (the ones that target S3 or NFS) are the off-box tier, and those are a different resource type entirely. Conflating the two is common enough that I'd call it the default misunderstanding. Longhorn Volume Health: The Gap Between 'Healthy' and Actually Working goes deeper on how the replica state and the dashboard's idea of "healthy" diverge. There's a capacity wrinkle too. Deleting a Longhorn snapshot does not immediately return the space. The snapshot gets marked for removal and the actual coalescing happens when the purge runs against each replica, so you'll see a volume that reports 40 GiB of usage against 12 GiB of live data for a while after cleanup. Recent Longhorn versions also cap snapshots per volume (250 by default), and a retain value on a recurring job that's too generous will hit that ceiling and start failing jobs. Retention that looked reasonable when the collection was small stops being reasonable at ten times the size. Application-level snapshots solve a problem block-level snapshots can't touch: portability. A Qdrant snapshot archive restores into any Qdrant instance, on any storage class, on any cluster, with a single API call: curl -X POST -H "api-key: $KEY " \ -F "snapshot=@/backups/2026-07-29/docs-1753747200.snapshot" \ "http://qdrant.example.local:6333/collections/docs/snapshots/upload?priority=snapshot" A Longhorn volume backup restores into a Longhorn cluster. That's a meaningfully smaller set of options on the day you're rebuilding. Distributed mode changes the math One caveat that catches people scaling from a single pod to a StatefulSet: in a distributed Qdrant deployment, a collection snapshot taken against one node contains only the shards that live on that node. Hitting the API through a Service round-robins your request to a random pod and gives you a partial backup that looks complete. For multi-node setups you need to enumerate the pods and snapshot each one, addressing them by their StatefulSet DNS names rather than the Service: for i in 0 1 2 ; do NODE = "qdrant- $i .qdrant-headless.vectordb.svc.cluster.local:6333" curl -sf -X POST -H " $AUTH " " $NODE /collections/ $C /snapshots?wait=true" done Restore is correspondingly per-node and needs the same shard topology on the target. If you're running distributed Qdrant as the memory layer for an agent system, that recovery procedure is worth writing down and rehearsing before you need it, which is exactly the kind of thing I end up doing in infrastructure and AI agent consulting work . Cleaning up what's already there If you're retrofitting this onto a running deployment, there's probably a pile of accumulated snapshots and a few orphaned PVCs. Find the snapshots first: kubectl exec -n vectordb deploy/qdrant -- \ sh -c 'du -sh /qdrant/snapshots && ls -1 /qdrant/snapshots | wc -l' Then the detached volumes, which are the ones nobody remembers creating: kubectl get pvc -A -o json | jq -r ' .items[] | select(.status.phase == "Bound") | "\(.metadata.namespace)/\(.metadata.name) \(.spec.resources.requests.storage)"' Cross-reference against pods that actually mount them. Anything bound with no consumer is either a deliberate cold spare or dead weight, and in my experience it's usually dead weight left behind by a Helm release someone uninstalled without --cascade . Snapshot the volume before you delete it, confirm nothing breaks for a week, then remove it. Space won't come back instantly for the reasons above. What I'd tell someone starting over Test the restore before you trust the backup. An untested backup is a hypothesis. Pick a snapshot at random once a month, upload it into a throwaway collection named restore-drill , run a search against it, compare the point count to the source, then delete it. Three minutes of work that converts a guess into a fact. Alert on backup freshness , not on job success. A CronJob that exits 0 while writing zero bytes is a green checkmark and an empty directory. The metric that matters is the age of the newest file on the NFS share, and if it exceeds two schedule intervals, page someone. Size the data PVC assuming snapshots will temporarily double your footprint, because they will. This is the single most common way a Qdrant deployment falls over, and the error you get back is generic enough that it sends people looking in entirely the wrong place. NFS is the middle tier, not the last one. It's off-box and it's simple, but it's still one server in one rack. The natural next step is pushing those archives to S3-compatible object storage, either through the same job or by pointing Qdrant's snapshots_config at S3 directly if your version supports it (check the docs for your specific release, this landed relatively recently and the config shape has moved). Velero + MinIO covers the object storage side of that, and it composes cleanly with everything above. The thing that stuck with me most from working through this: the Qdrant snapshot API is a perfectly good primitive that quietly assumes you've solved a problem it can't see. It writes a file. Where that file lands, how long it lives, and whether anything ever reads it back are decisions the API can't make for you, and they're the only decisions that determine whether you have a backup at all.

https://learn.finops.org/path/certified-finops-for-ai I was RIFed some months ago, working roles that were SAM/FINOPS related but not actually doing those things. I'd like my next role to be SAM/FINOPS but it's slow going with no tool certifications or practical experience. I got the FINOPS Cert Practitioner cert, which I understand is really viewed as somewhat "performative" when it comes to meaningful certifications. Would it be worthwhile going a little deeper with the FINOPS for AI cert? submitted by /u/RudeAd5133 to r/FinOps [link] [comments]

I am new to FinOps and Cloud Computing, i wanted to ask people who have experience in the FinOps space. Is there any financial modelling or specifically business case modelling done in FinOps? E.g. if there is any optimization opportunity or a new workload, is this a requirement from a CFO or board that they need to see a detailed financial model to show ROI and justify the spend? Reason I am asking is because I come from a core finance background just wanted to see if there is an overlap of my finance experience. submitted by /u/Appropriate_Class572 to r/cloudcomputing [link] [comments]

I’ve shared Kulshan here before, but we just pushed a pretty meaningful upgrade aimed at consultants doing AWS cost investigations. Or anyone (human or AI agent) trying to look at AWS cost and billing via a deterministic tool. The main addition is Consultant Evidence Export . The problem we were trying to solve is simple: an external consultant should not always need direct access to the customer’s AWS environment just to start investigating a billing issue. Now the customer can run Kulshan themselves, choose what they want to share, and generate a pseudonymized evidence package. They can scope it by date range, AWS accounts, services and tags, and optionally include Cost Explorer data. Kulshan can work from local CUR data or AWS CUR/Data Exports in S3. Account IDs, ARNs and resource identifiers are replaced with stable workspace-specific aliases, so the consultant can still follow the same account/resource through an investigation without seeing the original identifiers. We also added fail-closed checks before the package is created. Kulshan verifies the output schema, checks that the exported rows still match the scoped source data, and scans again for identifiers that should have been pseudonymized. If those checks fail, no ZIP gets created. The idea is basically: Customer keeps the credentials. Consultant gets the evidence. The package can help establish what changed, where, when, how much and which part of the environment was involved. The “why” still needs the people who understand the engineering and business context. Kulshan is still free and open source: https://github.com/MissionFinOps/kulshan Would be interested in feedback from consultants here: what additional evidence would you absolutely want included in a handoff like this? submitted by /u/MissionFinOps to r/FinOps [link] [comments]

I’m honestly a bit shocked by this. I deployed a pretty basic app last week, Next.js, Node, PostgreSQL and Redis. Nothing crazy. I went with AWS because I figured it would be easier to scale later, so I ended up with ECS/Fargate, RDS, ElastiCache, an ALB and NAT Gateway. The app has maybe 30–40 real users right now. Traffic is tiny. Then I checked the AWS billing dashboard this morning. $2,047 for the first 7 days. A lot of it seems to be infrastructure that’s basically sitting there, plus data transfer and NAT Gateway costs. I’m actively marketing the app, so moving everything to a basic VPS doesn’t really feel like a long-term solution if the traffic actually takes off. But obviously I can't keep spending thousands every week when I barely have users. So what are people actually using for this kind of setup? I’m looking for something that keeps costs reasonable while still being able to scale when traffic grows. Railway keeps coming up, along with VPS providers and other managed platforms. What would you use if you were starting this app again today? submitted by /u/OwlZealousideal4779 to r/aws [link] [comments]

There are only two of us handling FinOps for our whole cloud setup, so we're stretched thin and keep coming back to the idea of AI agents for rightsizing. Wondering if anyone here has actually leaned on them and whether it paid off. The pitch we keep hearing is that agents can watch utilization over time, flag over-provisioned VMs or containers, and recommend (or even auto-apply) the right size instead of us eyeballing dashboards once a quarter. Some tools claim they'll catch idle resources, downsize on a schedule, and factor in reserved instances or savings plans before suggesting a change. For a small team, that kind of always-on second set of eyes sounds like exactly what we need, but I want to know if the reality matches. What I'm trying to figure out: For those running this in production, are the recommendations actually good, or do they ignore context (spiky workloads, batch jobs, seasonal traffic) and try to shrink things that need the headroom? Do you let agents apply changes automatically, or keep a human in the loop for approvals? With only two of us, auto-apply is tempting but scary. Has anyone seen real savings, or does it mostly surface stuff you already knew? Any horror stories where an agent rightsized something into an outage? submitted by /u/ZeertY26 to r/AZURE [link] [comments]

Hey everyone, sorry if this is asked a lot already. Our AI spend is getting fragmented across multiple providers and multiple projects / agents. Would like to find a way that lets me track token spend attributed to these projects or API key for better visibility. Currently looking at dedicated management tool like Ramp's AI token spend management, or AI proxies / gateways to help me attach project metadata and enforce budget limits across all our provider keys. Would love to know what you guys are using and recommend, thanks! submitted by /u/sprogged to r/FinOps [link] [comments]

Hey everyone, I've been working on a tool called Plutus that pulls cost/event/usage data from cloud, AI, and SaaS providers into one dashboard so you can actually see why spend moved, not just that it did. I'm at the point where I'd like real users on it, but I want to be upfront about the current state. I've built and tested against every provider's API docs, but for a lot of the source list I don't have a live account of my own to validate against, so there are likely minor issues like fields mapped wrong, a sync that chokes on some edge case in a provider's real response, etc... I'm hoping a handful of people who already use one or more of the sources are willing to connect a read-only account and tell me what breaks. Signup is free and self-serve. If you hit anything broken, email [ [email protected] ](mailto: [email protected] ) with what you saw and I'll try and get it sorted ASAP. For your help, I'll upgrade you to a Growth-tier account for as long as you're actively testing. Just let me know your account email and I'll get your account upgraded as fast as I can. Any questions feel free to drop me a message on here or an email to the address above. You can find the marketing site at plutus-cloud.com or go straight to signup with console.plutus-cloud.com/ . submitted by /u/chenderson99 to r/FinOps [link] [comments]

I'm building CloudCostTree (estimates AWS costs from Terraform/CloudFormation, before you apply anything). A while back I added an opt-in flag that, instead of guessing, actually pulls real data from your AWS account: live Spot pricing, real CloudWatch CPU usage, volumes and IPs confirmed as orphaned. I'd tested it on its own, but never put the two reports side by side until today. Same file, same moment, nothing changed in between: Without real account data: 1 generic finding ("Graviton usually saves 20-40%"), score A (97/100). With real data: 7 findings, actual numbers. Spot price right now is literally $0.0082/hr vs $0.0208/hr on-demand, one instance averaging 3.4% CPU over 14 days, a volume and an Elastic IP confirmed orphaned via the API. Score C (76/100). The infra didn't get worse. What changed is whether the tool was actually allowed to look. Stuck with me a bit: a clean static report and a clean account aren't the same claim, and it's easy to mix them up until something forces the comparison. submitted by /u/Independent-Ease-609 to r/FinOps [link] [comments]

All of a sudden our .net code running in our k8s clusters cannot talk to our database. We just get connection failed errors. We have verified our database server is up and the dbs are online, and we also have not pushed any code since yesterday. Just wondering if anyone else is experiencing any sort of outage with azure submitted by /u/andrewsmd87 [link] [comments]

submitted by /u/Spiritual-Kitchen-79 to r/FinOps [link] [comments]

We have this recurring battle every single month. The FinOps side pulls a report of heavily over-provisioned instances sitting at like 15% average CPU, and we open Jira tickets to downsize them. Engineering almost immediately pushes back saying, "That's just an average. It spikes to 90% during our nightly batch jobs. If we downsize, the app is going to throttle and crash." The worst part is... they’re kind of right. We pushed a rightsizing recommendation through a while ago based on average utilization, hit a random traffic burst, and accidentally throttled production. Now, trust is totally broken, and engineering insists on over-provisioning everything "just in case." How are you guys bridging this gap in your orgs? Are you forcing everyone to use p95/p99 metrics before making a recommendation, or is there a better workflow to get devs to actually execute on these without the constant fear of breaking things? submitted by /u/CloudsAnalytics to r/FinOps [link] [comments]

Hey everyone, Most system design articles talk about architectures in pure theory ("add a CDN, add Redis, add Kafka"). But they rarely quantify: What does this actually cost per month on AWS at scale? At what RPS does the database connection pool saturate? How does adding Read Replicas affect the SLA nines? I modeled the Netflix Video Streaming Architecture in an interactive cloud simulator I have been building called ArchViz. Here is the breakdown: The Core Topology Edge / CDN: CloudFront distribution caching static assets & video chunks (~85% cache hit ratio). API Gateway + Auth: Microservices cluster running behind ALB on ECS Fargate. Persistence: Cassandra / DynamoDB for user viewing states + PostgreSQL for billing and account metadata. Event Streaming: Apache Kafka buffering real-time telemetry into AWS S3 cold storage. Simulated Metrics (at 50,000 RPS peak) Simulated AWS Monthly Cost: ~18,420/month(On−Demand)−>Dropsto 18,420/ month ( On − Demand )−> Dropsto 11,200 with 3-year Reserved instances & Spot worker nodes. Bottleneck Identified: The billing DB hits 92% CPU load without a Redis read-through cache layer when traffic spikes 3x. Security Scan: SOC2 warning triggered when the analytics S3 bucket lacked default KMS encryption. Interactive Model & Terraform Code If you want to play with the traffic sliders, simulate component failures, or export the production Terraform/Kubernetes code for this exact stack, you can test it directly in your browser: Live Interactive Canvas: https://archviz-studio.vercel.app Feedback on the FinOps pricing engine and IaC output is super welcome! submitted by /u/lexcodewell to r/aws [link] [comments]

Most cloud cost tooling I see is about usage: tags, SKUs, rightsizing, anomaly detection. The view I never had clean was the commitment as a financial object. The approved amount or the committed spend deal, drawn down by the actual invoices we booked, with a warning before we blew past it. It is the same problem I had running a services company, just a different bill. Usage or work runs past what was approved, the reconciliation happens late, and the overrun only shows up at close. World Commerce and Contracting pegs the leak on the contract side at roughly nine percent of value after signing, and committed cloud spend has the same shape. What I wanted was simple: The commitment, or a team budget, as a live account with a ceiling. Approved vs spent vs remaining, off the invoices you already book. An alert at a threshold so the true up or scope conversation happens while there is still room, not after. One number finance and engineering both trust, so there is no reconciliation fight at close, and clean showback by team or project. Not a replacement for your usage tool. More the layer above it: the money against the commitment, tied to your books. Honest disclosure: I built a tool that does exactly this, so I am biased. But I am genuinely curious how you all watch the commitment itself, not just usage. Spreadsheet, cost platform, something else? submitted by /u/FullBoatMain to r/FinOps [link] [comments]

Hey, We built ITS FINE : an AI-native platform that reconciles IT spend across systems that don't talk to each other: finance, procurement, assets, contractor tracking. It surfaces where money is leaking - duplicate licences, ghost contracts, auto-renewals nobody reviewed, contractor billing on closed budget, named by vendor, with amounts. We're opening beta slots. The trade is simple: What you Get - free, no commitment: - A written IT spend audit: every finding named by vendor, contract, and amount - Access to our full governance layer: anomaly detection, risk scoring, resolution workflows built in ; not just a report, an operating model for ongoing control If we find nothing, we tell you What we need: - Exports from your systems (resource management tool, finance, procurement, assets - whatever you have) - ~1 hour with someone who knows the data structure NDA signed before anything is sent. Masked vendors accepted. Best fit: 100–2,000 employees ; more than one system of record for cross-module detection. Happy to take questions here or by DM. submitted by /u/RMKH-95 to r/FinOps [link] [comments]

I’m researching how organisations handle the commercial side of cloud architecture changes after an initial cost baseline has already been approved. Suppose a team gets approval for an architecture based on a defined set of assumptions, then a few weeks later the design changes materially — larger database capacity, multi-region resilience, additional services, more storage, different traffic assumptions, etc. What actually happens in practice? Is the original baseline formally revised? Who decides whether the additional cost is acceptable? Is that decision part of the engineering/IaC approval process or handled separately? Do you retain both the original approved baseline and the revised one? Can you later tell which specific architecture decision caused a cost increase? How do you distinguish scope growth, design correction, usage growth and pricing variance? I’m particularly interested in real examples where the process worked well or broke down. Context: I work in project commercial management and I’m researching the boundary between engineering, FinOps and commercial governance. submitted by /u/Fit_Pound_3655 to r/FinOps [link] [comments]

Ran into a consulting client last month where the observability bill was over 40 percent of the total ops-tools line item. AWS shop, 15 microservices, Datadog for APM plus dashboards. The engineering lead had wired up a small internal agent to auto-triage CloudWatch alarms plus tag ownership, and it was catching most on-call pages before a human ever looked at the dashboard. The question their FinOps lead put on the table was: what is that Datadog premium actually buying us now? The APM traces still earn their line-item when the agent surfaces something and someone needs to dig, but the graph-watching use case was already gone. They are piloting a Datadog downgrade this quarter, keeping APM plus distributed tracing on the paid tier and moving the rest to Grafana over CloudWatch. Curious if anyone here has actually pulled this trigger on a real production line-item. Did the downgrade stick, or did the team end up putting some tools back? Second thing I keep hearing from FinOps practitioners: even if you drop dashboard SaaS spend, the engineering time saved on graph-staring seems to get eaten by tuning the agent that replaced it. Anyone tracking that as a real net-positive on hours, or is it a wash? submitted by /u/matiascoca to r/FinOps [link] [comments]

Just curious if anyone else is experiencing the same issue. Unable to send external or receive external. Seems Proofpoint might be down? submitted by /u/Dedicated__WAM to r/sysadmin [link] [comments]

Been looking at the new Cloud Billing Spend Caps preview since it landed last week and I keep going back and forth on whether I would actually flip it on in prod. Design is what it says on the tin. Set a hard cap per service, GCP stops new usage the moment it hits 100 percent, email alerts fire at 50, 80, and 100 percent. Fixed commitments like CUDs keep billing normally. What is stopping me is exactly what makes it useful. Hard-stop on a service that is silently in the critical path of something else feels like it wants a real dry-run first. Had a client last year where Vertex AI batch prediction was the fanout stage for a nightly job that fed three downstream systems. If a cap had fired at 3am, batch never completes, cascade of red dashboards on Monday morning. The alert would have been the fire, not the smoke. Curious what folks running this in preview have seen so far. Which services do you actually feel comfortable capping? Are you doing per-service caps or one umbrella per project? Anyone caught a real cost bug thanks to the 50 percent email or is it just noise once you have more than a handful of services? Not looking for the docs summary. Looking for the "here is where I stubbed my toe" version. submitted by /u/matiascoca to r/googlecloud [link] [comments]

Started tracking this after noticing our Claude usage for a multi-hour engineering session cost noticeably more than the size of the actual task should've justified. Went back through the session logs afterward to find where it actually went, and it wasn't one obvious spike, it was the same small pattern repeated across dozens of requests: full file contents resent every message even when two lines had changed, full conversation history replayed every turn instead of a compressed summary, full rewrites requested when a targeted diff would've done the job. None of that throws an error or shows up as a single line item. It just compounds quietly, message after message, until someone checks the usage dashboard and the number is higher than expected with no clear story for why. The part that feels familiar from a FinOps lens: this is structurally the same problem as unmonitored cloud spend before tagging and showback existed, cost accumulating because nobody's actively deciding what's worth paying for on each request, not because the underlying work got more expensive. Prompt caching (marking stable content so it's reused at a fraction of the cost) is the closest analogue to reserved capacity or committed use discounts, it only pays off if the cached prefix stays genuinely identical between calls, and a lot of teams break that silently by inserting a timestamp or per-user detail at the start of the block without realizing it kills the cache hit rate entirely. What's mostly missing right now, in my experience, is the equivalent of a FinOps practice applied to token spend specifically, an actual audit habit, not just a bigger budget line. Separating what's stable from what changes per request, summarizing instead of replaying full history, scoping context to what's relevant, and constraining output size all had a measurable effect once done deliberately instead of by habit. Wrote up the full breakdown with a before/after audit example here, disclosing that I'm the author: https://medium.com/@nagatomopedro05/the-hidden-cost-of-long-claude-sessions-2a6cc7655893 Curious if anyone here has actually folded LLM API spend into an existing FinOps practice, tagging, showback, budgets, or if it's still living entirely outside that process on most teams. submitted by /u/ClickOk5811 to r/FinOps [link] [comments]