I’m researching how organisations handle the commercial side of cloud architecture changes after an initial cost baseline has already been approved. Suppose a team gets approval for an architecture based on a defined set of assumptions, then a few weeks later the design changes materially — larger database capacity, multi-region resilience, additional services, more storage, different traffic assumptions, etc. What actually happens in practice? Is the original baseline formally revised? Who decides whether the additional cost is acceptable? Is that decision part of the engineering/IaC approval process or handled separately? Do you retain both the original approved baseline and the revised one? Can you later tell which specific architecture decision caused a cost increase? How do you distinguish scope growth, design correction, usage growth and pricing variance? I’m particularly interested in real examples where the process worked well or broke down. Context: I work in project commercial management and I’m researching the boundary between engineering, FinOps and commercial governance. submitted by /u/Fit_Pound_3655 to r/FinOps [link] [comments]
The cross-site community pulse: gold-layer posts + comment threads read live from the Communication Hub, ranked by importance. Turn a post into Discord / LinkedIn / X.
Ran into a consulting client last month where the observability bill was over 40 percent of the total ops-tools line item. AWS shop, 15 microservices, Datadog for APM plus dashboards. The engineering lead had wired up a small internal agent to auto-triage CloudWatch alarms plus tag ownership, and it was catching most on-call pages before a human ever looked at the dashboard. The question their FinOps lead put on the table was: what is that Datadog premium actually buying us now? The APM traces still earn their line-item when the agent surfaces something and someone needs to dig, but the graph-watching use case was already gone. They are piloting a Datadog downgrade this quarter, keeping APM plus distributed tracing on the paid tier and moving the rest to Grafana over CloudWatch. Curious if anyone here has actually pulled this trigger on a real production line-item. Did the downgrade stick, or did the team end up putting some tools back? Second thing I keep hearing from FinOps practitioners: even if you drop dashboard SaaS spend, the engineering time saved on graph-staring seems to get eaten by tuning the agent that replaced it. Anyone tracking that as a real net-positive on hours, or is it a wash? submitted by /u/matiascoca to r/FinOps [link] [comments]
Proofpoint outage
by Dedicated__WAM
Just curious if anyone else is experiencing the same issue. Unable to send external or receive external. Seems Proofpoint might be down? submitted by /u/Dedicated__WAM to r/sysadmin [link] [comments]
Been looking at the new Cloud Billing Spend Caps preview since it landed last week and I keep going back and forth on whether I would actually flip it on in prod. Design is what it says on the tin. Set a hard cap per service, GCP stops new usage the moment it hits 100 percent, email alerts fire at 50, 80, and 100 percent. Fixed commitments like CUDs keep billing normally. What is stopping me is exactly what makes it useful. Hard-stop on a service that is silently in the critical path of something else feels like it wants a real dry-run first. Had a client last year where Vertex AI batch prediction was the fanout stage for a nightly job that fed three downstream systems. If a cap had fired at 3am, batch never completes, cascade of red dashboards on Monday morning. The alert would have been the fire, not the smoke. Curious what folks running this in preview have seen so far. Which services do you actually feel comfortable capping? Are you doing per-service caps or one umbrella per project? Anyone caught a real cost bug thanks to the 50 percent email or is it just noise once you have more than a handful of services? Not looking for the docs summary. Looking for the "here is where I stubbed my toe" version. submitted by /u/matiascoca to r/googlecloud [link] [comments]
Started tracking this after noticing our Claude usage for a multi-hour engineering session cost noticeably more than the size of the actual task should've justified. Went back through the session logs afterward to find where it actually went, and it wasn't one obvious spike, it was the same small pattern repeated across dozens of requests: full file contents resent every message even when two lines had changed, full conversation history replayed every turn instead of a compressed summary, full rewrites requested when a targeted diff would've done the job. None of that throws an error or shows up as a single line item. It just compounds quietly, message after message, until someone checks the usage dashboard and the number is higher than expected with no clear story for why. The part that feels familiar from a FinOps lens: this is structurally the same problem as unmonitored cloud spend before tagging and showback existed, cost accumulating because nobody's actively deciding what's worth paying for on each request, not because the underlying work got more expensive. Prompt caching (marking stable content so it's reused at a fraction of the cost) is the closest analogue to reserved capacity or committed use discounts, it only pays off if the cached prefix stays genuinely identical between calls, and a lot of teams break that silently by inserting a timestamp or per-user detail at the start of the block without realizing it kills the cache hit rate entirely. What's mostly missing right now, in my experience, is the equivalent of a FinOps practice applied to token spend specifically, an actual audit habit, not just a bigger budget line. Separating what's stable from what changes per request, summarizing instead of replaying full history, scoping context to what's relevant, and constraining output size all had a measurable effect once done deliberately instead of by habit. Wrote up the full breakdown with a before/after audit example here, disclosing that I'm the author: https://medium.com/@nagatomopedro05/the-hidden-cost-of-long-claude-sessions-2a6cc7655893 Curious if anyone here has actually folded LLM API spend into an existing FinOps practice, tagging, showback, budgets, or if it's still living entirely outside that process on most teams. submitted by /u/ClickOk5811 to r/FinOps [link] [comments]
This article explains how to build and test a cross-cloud currency agent. An Amazon Bedrock master agent , built with Strands Agents and hosted on Amazon Bedrock AgentCore Runtime in AWS us-east-1 , delegates to a Google ADK worker on GCP Cloud Run in us-central1 over A2A v1.0 . The master cross-checks the worker against an MCP exchange-rate tool and measures the latency, reliability, and failure behavior of cross-cloud verification. What is this project trying to do? Most Agent-to-Agent (A2A) protocol demos stop at "look, the HTTP 200 OK request succeeded." That is a smoke test, not an interoperability benchmark. This project goes further: the Bedrock master owns the user interaction and benchmark policy. It discovers and calls a Google ADK worker running on GCP Cloud Run, then compares the worker result with a local MCP stdio exchange-rate tool backed by live Frankfurter daily reference rates. We also compare the performance, developer experience, and wire compatibility with a previous benchmark run using Microsoft Foundry in Azure ( gpt-5-mini ). Together, the runs cover components hosted across AWS, Azure, and GCP. The benchmark addresses four questions: Can an AgentCore-hosted Bedrock master discover and invoke a Google ADK worker through an A2A agent card with no framework-specific glue? What latency and token overhead does remote-agent verification add? Does independently verifying an MCP tool result over A2A improve correctness or failure recovery enough to justify that overhead? Which measurements are portable across coordinators, and which require a fully hosted AgentCore benchmark run? Reusing the original currency agent This builds directly on the currency agent from the previous articles in this series: Getting Started with MCP, ADK and A2A | Google Codelabs GitHub - jackwotherspoon/currency-agent That agent — built with Google ADK, Gemini 2.5 Flash, and a FastMCP exchange-rate server backed by the free Frankfurter API — serves as the remote worker and independent verifier in this project. The new repository adds the AgentCore coordinator and benchmark suite: GitHub - xbill9/bedrock-adk-a2a-currency Architecture CLI / Boto3 Test Runner (AWS SigV4 Auth) | Bedrock AgentCore Runtime hosted master (AWS, us-east-1, Amazon Nova Micro) Strands Agents orchestration | +-- MCP stdio --> Frankfurter rates (in-container stdio process) | +-- A2A v1.0 --> Cloud Run (GCP, us-central1) | Google ADK worker (gemini-2.5-flash) | MCP HTTP --> Frankfurter rates The Bedrock master answers every conversion request through three distinct evaluation modes: Mode What happens Why it exists mcp_only Bedrock master calls the local MCP rate tool Baseline single-agent performance a2a_only Bedrock master delegates to the GCP ADK worker over A2A v1.0 Measure remote-agent behavior and network latency verified MCP result independently checked against the remote ADK agent over A2A Measure the accuracy-versus-overhead tradeoff Both sides read the same Frankfurter daily reference rates on purpose: when the two clouds disagree, that measures protocol, model, and orchestration behavior, not data-source skew. Rule one: the model never does math Currency conversion is a poor job for an LLM and a good job for Python's Decimal . The domain layer is framework-independent and uses Pydantic models. Numeric agreement is evaluated in code using relative difference; no LLM is asked, "Do these numbers look close to you?" difference = abs ( primary . converted_amount - verifier . converted_amount ) relative = difference / abs ( primary . converted_amount ) agreed = relative <= tolerance # default 0.005 (0.5%) The failure policy is explicit rather than emergent: MCP fails, A2A succeeds → return the remote result, labeled unverified . A2A fails, MCP succeeds → return the tool result with a "verification unavailable" warning. Both succeed but disagree → return both quotes and issue a warning; never silently pick the LLM's preferred rate. Both fail → return a strongly typed failure ( validation , provider , authentication , transport , timeout , protocol ); never fabricate a rate. Because "which layer broke" is a core research question, every adapter exception is normalized into exactly one typed failure at the boundary. The wire mismatch: A2A v0.3.0 vs. v1.0 The first attempt to connect the AgentCore coordinator to the Google ADK currency agent died immediately on invocation: a2a.utils.errors.MethodNotFoundError: Method not found Observed root cause: a protocol-version mismatch between A2A v0.3.0 and v1.0, with no automatic fallback negotiation in the tested client. The modern A2A client ( a2a-sdk>=1.0 ) calls the A2A v1.0 JSON-RPC method SendMessage . Older ADK agents ( a2a-sdk 0.3.x ) only expose the v0.3.0 method message/send . The client fetched the agent card — which explicitly declared protocolVersion: 0.3.0 — but attempted the v1.0 method anyway. The initial ecosystem package pins were also mutually exclusive: Package a2a-sdk Requirement Status strands-agents 1.50.2 >=1.0.0,<2 Compatible google-adk 2.1.0 – 2.4.0 >=0.3.4,<0.4 Incompatible google-adk 2.5.0 >=0.3.4,<2 Compatible ✅ a2ui-agent-sdk (through 0.4.0) <0.4.0 Incompatible ❌ google-adk 2.5.0 updated its dependencies to support a2a-sdk 1.x . However, A2UI extensions currently pin the older v0.3.0 protocol. For this benchmark, A2UI was omitted so both AWS and GCP sides could operate on A2A v1.0 ( a2a-sdk 1.1.2 ) . Hosting the Bedrock master on Amazon Bedrock AgentCore Deploying the master to Amazon Bedrock AgentCore Runtime involved navigating several fast-moving SDK and platform details observed during our build on 2026-07-28: 1. Model selection: Anthropic access requirements vs. Amazon Nova Micro In the account used for this build, Anthropic models such as Claude 3.5 Sonnet required a one-time use-case submission ( PutUseCaseForModelAccess ) and an AWS Marketplace subscription agreement. To keep the setup automated, we configured the coordinator to use Amazon Nova Micro ( us.amazon.nova-micro-v1:0 ). Nova Micro required no approval form in our test account, supported native tool calling in the tested workflow, and produced subsecond model responses. 2. Inference profile IDs In our deployment, using the bare model ID ( amazon.nova-micro-v1:0 ) returned an HTTP 400 ValidationException requiring on-demand throughput configuration. Passing the regional inference profile ID ( us.amazon.nova-micro-v1:0 ) resolved the error. 3. CLI tooling transition The older Python pip -based starter toolkit ( agentcore configure / agentcore launch ) was deprecated in June 2026. Deployment now uses the official @aws/agentcore npm CLI (Node 20+, CDK-based). Coordinator entry point (abridged from app/CurrencyCoordinator/main.py ) from bedrock_agentcore.runtime import BedrockAgentCoreApp from strands import Agent , tool from coordinator.hosted_tool import run_currency_benchmark from model.load import load_model app = BedrockAgentCoreApp () tools = [ tool ( run_currency_benchmark )] # The full source defines a bounded, session-scoped agent factory here. @app.entrypoint async def invoke ( payload , context ): session_id = getattr ( context , " session_id " , " default-session " ) agent = get_or_create_agent ( session_id ) prompt = payload . get ( " prompt " , payload . get ( " messages " , "" )) result = await agent . invoke_async ( prompt ) return { " result " : str ( result )} if __name__ == " __main__ " : app . run () The hosted runtime also fails closed when its GCP worker is missing: { "name" : "CURRENCY_REQUIRE_GCP_ADK" , "value" : "1" } With that setting, a2a_only and verified return gcp_adk_not_configured if CURRENCY_A2A_ENDPOINT is absent. A deployment can no longer appear to exercise A2A while silently using a local fixture. The Bedrock model configuration also sets BEDROCK_MAX_TOKENS=1024 explicitly to bound output and quota usage. The Google side: ADK on Cloud Run The remote verifier container colocates two processes: the FastMCP Frankfurter server on localhost and the A2A app listening on $PORT . Gemini API keys are retrieved securely from GCP Secret Manager: gcloud secrets create gemini-api-key --data-file = " $HOME /gemini.key" gcloud run deploy currency-adk-a2a \ --source adk_agent --region us-central1 \ --allow-unauthenticated --min-instances = 0 --max-instances = 2 \ --set-secrets "GOOGLE_API_KEY=gemini-api-key:latest" \ --set-env-vars "MCP_SERVER_URL=http://127.0.0.1:8081/mcp,GENAI_MODEL=gemini-2.5-flash" Setting --min-instances=0 allows Cloud Run to scale to zero when idle. The coordinator's timeout is set to 60 seconds to accommodate initial container cold starts. How to run the benchmark The repository includes a complete local test suite that runs deterministically without credentials or cloud infrastructure: # 1. Clone & install dependencies git clone https://github.com/xbill9/bedrock-adk-a2a-currency cd bedrock-adk-a2a-currency pip3 install --user -e ".[dev]" # 2. Run unit and integration tests (deterministic fixtures) pytest # 3. Test local CLI modes currency-benchmark 100 USD CAD EUR --mode mcp_only currency-benchmark 100 USD CAD EUR --mode verified --transport mcp-stdio # 4. Execute full evaluation matrix currency-evaluate --output /tmp/currency-results.jsonl --summary /tmp/currency-summary.json To deploy and test the hosted Bedrock master: ./infra/sync_app.sh agentcore deploy -y agentcore invoke "Convert 100 USD to EUR and CHF in verified mode." Hosted smoke test: Bedrock master → GCP ADK worker On 2026-07-29, I deployed the updated master to AgentCore Runtime in us-east-1 and invoked all three modes through the hosted InvokeAgentRuntime API: Hosted mode Observed result mcp_only HTTP 200; live mcp-stdio:frankfurter-live quote a2a_only HTTP 200; live gcp-adk-a2a-worker quote verified HTTP 200; MCP and GCP ADK agreed exactly for EUR and CHF The verified request converted 100 USD to EUR and CHF. The deterministic comparison recorded relative_difference: "0" and agreed: true for both currencies, with no failures or warnings. The benchmark tool completed in approximately 3.08 seconds. This was an end-to-end smoke test, not a full hosted latency distribution. It exercised the complete path: AWS SigV4 invocation → AgentCore Runtime → Nova Micro tool selection → MCP stdio / Frankfurter → A2A v1.0 → GCP Cloud Run → Google ADK / Gemini → deterministic Decimal comparison The smoke test also found a real orchestration bug. On the first request, Nova Micro read “Convert 100 USD to EUR” but claimed the target currency was missing and asked the user to confirm it. The master prompt now includes an explicit natural-language parsing rule and forbids confirmation requests for information already present. After redeployment, the same request called the benchmark tool directly. A regression test preserves that behavior. Cross-cloud benchmark results We executed the 38-case evaluation matrix across all three modes: 114 records per run. The 2026-07-28 warm run exercised the framework-independent coordinator locally against the live GCP Cloud Run ADK endpoint; it did not measure the AgentCore hosting layer. The 2026-07-27 run is the retained Azure-era baseline. Keeping those labels explicit avoids attributing local harness latency to AgentCore. Observed run Evaluation mode Success rate Median latency p95 latency Agreement rate 2026-07-28 warm local harness → GCP mcp_only 100% 286 ms 540 ms N/A 2026-07-28 warm local harness → GCP a2a_only 100% 2.09 s 6.10 s N/A 2026-07-28 warm local harness → GCP verified 100% 1.87 s 4.33 s 96.77% 2026-07-27 Azure-era baseline → GCP mcp_only 100% 297 ms 1.09 s N/A 2026-07-27 Azure-era baseline → GCP a2a_only 100% 1.69 s 4.82 s N/A 2026-07-27 Azure-era baseline → GCP verified 100% 1.71 s 4.15 s 96.77% Key findings The live protocol path was reliable: the warm 2026-07-28 run completed all 114 records successfully. Fault-injection cases are included in the aggregate, so agreement rate is not expected to be 100%. Concurrent execution limits verification overhead: verified-mode latency is dominated by the remote A2A round trip rather than the sum of MCP and A2A latency. Hosted AWS → GCP interoperability was observed: all three modes completed through AgentCore. The verified EUR and CHF quotes had zero relative difference, no failures, and no warnings. This remains a smoke-test result, not a 114-record hosted latency distribution. Hosted performance remains to be measured: token usage, cost, and repeated warm/cold AgentCore distributions are still open benchmark work. Lessons learned Check A2A SDK major versions first: A2A v0.3.0 ( message/send ) and v1.0 ( SendMessage ) are wire-incompatible. If you see MethodNotFoundError , inspect the a2a-sdk version on both client and server before debugging prompt logic. Use inference profile IDs on Bedrock: In our hosted deployment, the regional inference profile ID ( us.amazon.nova-micro-v1:0 ) avoided the on-demand throughput error returned for the bare model ID. Account for remote cold starts: A 10-second client timeout worked locally, but the Cloud Run scale-from-zero path needed a longer window. We used 60 seconds for this benchmark. Keep math out of the prompt: Deterministic Python Decimal arithmetic prevents LLM calculation errors from affecting conversion and agreement checks. The checks therefore measure differences in returned results, not the model's arithmetic ability. A2A verification provides independent fault detection: The faster mcp_only path is useful as a baseline, while cross-cloud A2A verification adds an independent result for failover and anomaly detection. Whether the overhead is justified depends on the workload. Test natural-language argument extraction: Tool availability is not enough. The master model can still fail before invocation by misreading an argument that is plainly present. Keep a hosted smoke case for natural-language parsing, not only structured tool calls. Repository and source code The complete benchmark codebase, deployment scripts, test suite, and raw evaluation datasets are available on GitHub: GitHub - xbill9/bedrock-adk-a2a-currency If you are building multi-cloud agent systems with Amazon Bedrock AgentCore, Google ADK, or Microsoft Agent Framework, feedback and benchmark contributions are welcome.
Our FinOps tool is blind to AI spend and up for renewal. Anyone switched to PointFive?
by Flateland-Chio
Pulled our numbers for the renewal and the AI and GPU line is about 4x last year, but our cost tool drops nearly all of it into one 'other' row I can't split by team or even by service. Good on EC2 and RDS, useless on the part that's growing fastest. PointFive is the name three separate people pointed me to for this, mainly because it doesn't treat the AI and GPU spend as a footnote. Sat through the demo, looked good, they always do. The one thing a demo can't answer, when it flags waste is it stuff my current tool already nags me about or does it surface things we didn't know were running. If someone's live on it and it found waste their old tool was blind to, that's the only reason I'd rip out a renewal to switch. submitted by /u/Flateland-Chio to r/FinOps [link] [comments]
Financial Modelling in FinOps/Cloud Investments
by Appropriate_Class572
I am new to FinOps, i wanted to ask people who have experience in the FinOps space. Is there any financial modelling or specifically business case modelling done in FinOps? E.g. if there is any optimisation opportunity or a new workload, is this a requirement from a CFO or board that they need to see a detailed financial model to show ROI and justify the spend? Reason I am asking is because I come from a core finance background just wanted to see if there is an overlap of my finance experience. submitted by /u/Appropriate_Class572 to r/FinOps [link] [comments]
O365 Outage
by AncientVase
Is anyone else seeing these issues. Just got a call from Help Desk to check it out. Sharepoint home pages are accessible but no files are. Down detector shows a spike but only 128 reports so far. Central/East US region. submitted by /u/AncientVase to r/sysadmin [link] [comments]
submitted by /u/rushi2105 to r/FinOps [link] [comments]
Running an interview series on data and AI compute cost, looking for practitioners
by Sad-Fishing-7666
Disclosure: I work with Haevek, a data compute platform. This isn't a pitch and there's nothing to sign up for. Flagging it because this sub asks you to declare. We're putting together a community content series where we interview people who build and run modern data and AI infrastructure. No product talk, no script, no gated landing page. Conversations that get published for whoever finds them useful and nothing gets published without your review. Our view, which you're welcome to tear apart: the bill isn't the software, it's the infrastructure to run it. Open source compute is free to license, but the clusters stay on and consumption pricing climbs with every workload, so cost grows faster than the value coming back. We think the fix is a more efficient engine, not a bigger budget. Plenty of people disagree, which is usually the more interesting conversation. Topics people have picked so far: Where data and AI compute cost actually goes, and why the bill keeps growing as teams do more Scaling AI and agent workloads, where the limit is the cost of running inference over and over rather than the model or the talent What teams get wrong about controlling data and AI cost? Who I'm hoping to talk to: Director or Head of FinOps, Head of Cloud Cost / Cloud Economics VP or Head of Data Platform / Data Engineering who's had the cost conversation forced on them anyone who's actually cut a big data or AI compute line item and can explain what they did Company-wise, anywhere the bill is big enough to be political. Enterprise, scale-up, public sector, doesn't matter. 30 minutes, remote, you get the recording and can cut clips from it. If that's you, or you know someone, DM me. submitted by /u/Sad-Fishing-7666 to r/FinOps [link] [comments]
Hi everyone, I would like to hear how experienced DevOps engineers approach monitoring for large public-facing applications. We have a .NET e-commerce platform with: - ASP.NET Core MVC + Angular - SQL Server - Elasticsearch (~10M products) - RabbitMQ - IIS hosting - Multiple public domains/subdomains - Heavy SEO crawling and unknown bots One thing we learned is that monitoring only CPU, memory, and disk is not enough. We have experienced situations where: - CPU and RAM looked normal, but the application was slow - The server was reachable, but users experienced downtime - TCP exhaustion caused issues - Elasticsearch had problems affecting search performance - Bots generated a lot of unnecessary traffic - Slow requests were not obvious from infrastructure metrics I would like to know what metrics and alerts you consider essential for this type of system. Some things I think are important: Application level: - Request rate (RPS) - Response time (p50/p95/p99) - HTTP status codes (4xx/5xx) - Slow endpoints - Exception rate - Thread pool starvation - GC pauses - .NET runtime counters - Memory allocations IIS / Web server: - Current connections - Request queue length - Worker process health - Application pool recycling - Failed requests - Connection errors Network: - TCP connections - TIME_WAIT count - Connection failures - Bandwidth usage - Top clients/IPs - Suspicious user agents Elasticsearch: - Cluster health - JVM memory pressure - Heap usage - Search latency - Query failures - Slow queries - Unassigned shards - Disk usage SQL Server: - CPU - Blocking queries - Deadlocks - Query duration - Connection pool usage - Wait statistics RabbitMQ: - Queue length - Consumer count - Message processing time - Dead letters - Memory usage Security / traffic: - Requests to suspicious paths: - /.env - /.git - wp-admin - Bot traffic percentage - High-frequency clients - Rate limit violations My question: If you were responsible for operating a public .NET application like this, what dashboards and alerts would you consider mandatory? Also, what are some metrics you discovered were extremely valuable only after a production incident? I am especially interested in real-world experience rather than a theoretical checklist. Thanks! submitted by /u/No-Card-2312 to r/devops [link] [comments]
CloudCostTree can now auto-apply the safe FinOps recommendations it finds (--optimize)
by Independent-Ease-609
New feature I shipped: --optimize takes the FinOps recommendations CloudCostTree already shows you and applies the ones that are safe to apply mechanically, no architecture or availability trade-off involved. What gets auto-applied: gp2 to gp3 (EBS and RDS storage), provisioned IOPS to gp3 where it's not needed, previous-generation instance types to current-gen, RDS backup retention capped at 30 days, DynamoDB provisioned to on-demand, non-production resources rescheduled to business hours. On Pro with --with-usage, also confirmed orphaned EBS volumes and snapshots, empty target-group load balancers, and unassociated Elastic IPs. What it will never auto-apply: x86 to Graviton (changes CPU architecture), removing Multi-AZ (changes your failover story), CPU or memory based right-sizing from real usage data (measured but still inferred). Those stay as suggestions you confirm yourself. Screenshots show the flow in the VS Code extension against a small test file: the two safe recommendations it found (save $7.01/mo switching off a previous-gen instance, $4.00/mo off a gp2 volume), picking which to apply, and the result, total down from $87.97 to $76.97/mo and the Cost Score up from B to 88. Free tier, and the same thing works from the CLI with cloudcosttree analyze --optimize. https://cloudcosttree.com submitted by /u/Independent-Ease-609 to r/FinOps [link] [comments]
submitted by /u/PersonaSeria to r/FinOps [link] [comments]
We run a SaaS company in Brazil. Our entire production stack sits in AWS account 393686273302: EKS, RDS, ElastiCache, SES, S3. Application code, customer data, payment records. We think a card change triggered the flag. When we opened the account, the agency that builds our platform (Specter) registered one of their corporate cards so they could provision infrastructure while we set up our own payment method. When the first invoice came due, we replaced that card with our company card and paid the invoice in full. AWS restricted the account after that payment cleared. Our billing console shows R$ 0.00 outstanding today. No open invoice. The suspension notice says non-payment. We have fought verification flags on this account since July: - AWS denied two EC2 vCPU quota increases (L-1216C47A on-demand, L-34B43A08 spot), then granted them after we appealed. - AWS denied SES production access in case 178458595700478, then granted it after we appealed. - CloudFront returned 403 "verification required" and the flag never cleared. - On August 10, RunInstances began returning "This account is currently blocked and not recognized as a valid account". CreateFleet returned MaxFleetCountExceeded while we ran zero fleets. Our Spot quota read 0. Our MediaConvert queue flipped to PAUSED and UpdateQueue returned Forbidden. We opened case 178638951500949 on August 10 and wrote in it that our launch was the next day. We opened a second case; AWS closed it as a duplicate and pointed us back to the first. Nobody from the verification team wrote to us. On August 11st, AWS suspended the account. AWS Health reports our EKS cluster kloel-eks-prod as IMPAIRED: "We couldn't assume the Amazon EKS cluster management service-linked-role" and "We couldn't find or access the AWS KMS key associated with your cluster", with a warning that the control plane shuts down in two days. The NLB in front of our API stopped accepting connections, so api.kloel.com and checkout-api.kloel.com time out from every network we tested. IAM keys that worked before the suspension now return InvalidClientTokenId, so we cannot read our own resources or export a backup. We processed real customer payments hours before the suspension, card and PIX. Those customers now hit a dead platform. We opened case 178647610300640 and uploaded every document AWS requested through their verification link the same day. Nobody has answered us since. Support answered that case with this: > "As this particular inquiry is handled by one of our program support teams, I've forwarded your case directly to them. A member of this team will be in touch with you soon. Our program support team can only communicate through email." We received the same sentence on the earlier case, and nobody contacted us after it. We scheduled our launch for August 11. It did not happen. We are fielding questions from investors and partners about why the product we demoed to them is unreachable. Every minute our platform remains unreachable, the pressure over our company and workers grow, if this goes any further the damage can be inestimable. Two questions for anyone who has been through this: Is there a way to reach a human who can review a payment and reinstate an account? Support forwards our cases and nobody answers. How do we get written confirmation from AWS that our RDS database, EBS volumes and snapshots stay intact while the review runs? We paid the invoice, replaced the card, sent the documents and opened the cases. If anyone from AWS reads this: account 393686273302, cases 178647610300640, 178638951500949 and 178458595700478 have the full history, and we will send anything else you need by DM. submitted by /u/Pandowso to r/aws [link] [comments]
Something easy to miss when using cost tooling: it's most useful at two separate moments, not just once before a deploy
by Independent-Ease-609
Before deploy: running the analysis against your IaC files up front shows you the full cost breakdown plus savings recommendations, and applies whatever's safe to apply without a human decision, before anything actually gets provisioned. After deploy: once it's live, re-running the same analysis with real CloudWatch usage data (via your own read-only AWS credentials) refines those recommendations against actual utilization instead of static config assumptions. The reason this matters for FinOps specifically: static config tells you what something was provisioned for, not what it's costing you in practice. A right-sizing call made purely from declared instance types will miss real idle capacity, and one made purely from live usage misses waste that never should've been provisioned in the first place. Catching both requires checking at both points in the lifecycle, not just once. (Built this into CloudCostTree, a CLI I've been working on, happy to go into specifics if useful.) submitted by /u/Independent-Ease-609 to r/FinOps [link] [comments]
My CFO asked me to break our AI spend down by team and I couldn't do it.
by Dalius-Gabryelle
Our CFO caught me after standup and asked a totally fair question, how much is each team spending on all this AI stuff. I said I'd have a number by Friday. Took me until the following Wednesday to admit I couldn't. I'd assumed it would be like AWS where I can slice spend by team in a few clicks, atleast we tagged everything. Then opened the billing expecting the some breakdown but it's just a big monthly number and a graph that goes up. One team lead keeps insisting their usage is not that substancial which without the numbers i cant prove otherwise. We'd handed his squad a shared API key so their spend was all piled onto one. Now, even if I nailed the attribution, most of an agent's bill is the framework re-sending its whole setup every turn, not anything the dev ever typed. I'd be walking into a room to bill someone for tokens they never wrote and can't even see. There's a paper going round with the numbers, arxiv 2607.12161. Anyway. I still owe her that spreadsheet. submitted by /u/Dalius-Gabryelle to r/FinOps [link] [comments]
Doing some research around cloud and AI/token commitment economics and helping NGEN gather feedback on the model. Curious to get the FinOps community’s perspective. The idea is to aggregate compute/token demand across companies, negotiate larger commitments with providers, and use prepayment/financing to offer better pricing and more flexibility. A few things I’m curious about: How much additional savings would make this worthwhile — 5%? 10%+? Is commitment flexibility potentially more valuable than additional savings? Does this make more sense for mid-market companies that don’t already have significant negotiating leverage? NGEN is also collecting anonymous, non-binding indications of demand here (takes ~1 min, no commitment/signature): https://www.ngencompute.com/indication Would genuinely love to hear why you think this would or wouldn’t work. submitted by /u/melc10 to r/FinOps [link] [comments]
Bookkeeping & FinOps as a service!!
by FirstMechanic2151
I’m 22 studying engineering final year. Planning to start out a service business online. I’m looking into providing bookkeeping & FinOps as a service for creative/marketing agencies & eCom businesses Any advise on prerequisites of providing this service from experts in this field could be of help. Thanks, appreciate your time:) submitted by /u/FirstMechanic2151 to r/FinOps [link] [comments]
submitted by /u/Bartaseth to r/FinOps [link] [comments]