CommPulse

Anybody here who feels AI SRE is a gimmick? I mean, what are we doing with this, honestly? Not gonna name names, but our observability platform ships one, and it takes 10–12 minutes to run an incident analysis and comes back with "hey, these are 2 things that are broken, and these are 5 probable causes." In the last three incidents, the causes it flagged were real issues acting up as bugs, but not related to the current incident at all. So I'm getting a faster RCA, just a wrong one. I raise a ticket and get misleading suggestions. Just to get an unbiased opinion, I tried out a few other names on the market: similar results. Happy to hear if anyone has had a different experience. submitted by /u/Less_Ad8195 to r/devops [link] [comments]

We're seeing issues with Ubuntu clients (22.04/24.04/26.04) failing to check in with Intune since thursday. Intune logs on the clients say "Failed to Check in with Intune. Failed to update device inventory. <some MS URL> Status Code 500". Some other logs also say 400 or 404. Latest version of intune-portal and microsoft-identity-broker installed. Downgrade did not help. Trying to enroll a new device will either: result in the device only landing in Entra, but not in Intune result in the device appearing in both Entra and Intune, but without any inventory data Entra shows "Ubuntu+26.04+LTS" as OS version, while Intune shows either nothing (blank) or "0.0.0.0", resulting in the device not picking up our compliance policy -> device will be not compliant -> user cant login to M365 apps. dsreg on the client shows that it correctly registered to Entra. Trying to trigger a sync or log in locally in the Intune portal app will immediately result in an error message ("Device could not be registered. There was an error, please try again"). Existing clients that are already enrolled also fail checkin and logon on the Intune portal app. But since they're still compliant users can still log in to M365 apps. There seems to be an issue going on at Microsoft's side. There's an open issue in the admin center (OP1459987), which sounds like it might be related giving the intune-portal app fails to update its device inventory. "Some admins may see outdated device inventory and update status information in the Microsoft 365 Apps admin center" submitted by /u/EpicSimon to r/sysadmin [link] [comments]

We have a currently CSP providing us licensing with a 10% discount over Microsoft's direct pricing, and thats fine by us. This relationship started out great with having a dedicated person we could ask licensing questions to that was fairly knowledgeable, so that was a nice perk when navigating the licenses available became a bit too much. Unfortunately we've began running into billing issues. These issues are taking far too long to work out. Our contacts have also become fairly unresponsive to our emails taking weeks to get back to us at times. This all prompts us to start looking around at options. Does anyone have any companies they currently have a good experience with? I don't really want to cold call, find a CSP that promises everything, and then puts us right back where we're at right now. We're just looking for licensing with a discount over going direct via Microsoft, this should be purely transactional so we need someone that makes it that easy. We are in the ~150 user range since I know that can make a large difference. Located west coast US. Thanks! submitted by /u/FIRE0118999881999119 to r/sysadmin [link] [comments]

submitted by /u/WancloudsInc to r/FinOps [link] [comments]

I'm sitting here scratching my head trying to figure this one out, I've went through our group policy settings multiple times and I can't find anything I might have configured that would cause this problem. new out of the box lenovo thinkpad laptop with 25H2 preload on it from factory. works great until it is joined to the domain, as soon as it joins the domain and rebooted, subsequent logins for any domain account or local account on the machine are having issues with not being able to run any apps other than edge or file explorer and on login it's giving an error stating that your system administrator has blocked the program. when I check the logs I'm seeing DistributedCOM errors with event id 10001 for Microsoft.AADBrokerPlugin, that appear to be related to windows security core background get token task classid webaccountprovider being unavailable. I'm not sure what the heck is going on here, but I need to get it fixed before it spreads to any of our existing windows 11 machines if it was caused by a malfunctioning windows update or something else. I'm about to blow the machine away and just load a clean install of 24h2 on it, but if anyone knows how to go about fixing this I'd like to try that first before I give up. this one is a replacement laptop for an employee and he can manage for a few days with his current laptop. my google-fu skiils haven't came up with anything that has worked thus far on it. I've reset it and it runs fine again until it is joined to our domain. I did notice when I ran systeminfo from a command prompt it is reporting that App Control for Business policy is enabled and app control for business user mode policy is set to audit. Looking on another machine that is still working fine, those two settings aren't activated. I tried local group policy to turn device guard off to see if that made a difference, but it didn't. submitted by /u/doctorevil30564 to r/sysadmin [link] [comments]

I currently handle SaaS & AI procurement for client orgs and want to add FinOps to my skillset. I'm non-technical, was in sales before procurement, and have a decent grasp on financial concepts having passed CFA level I (from a previous career path I thought I would go down but didn't). I have minimal cloud knowledge, should I start with AWS CCP to understand cloud fundamentals and so better understand the scenarios FOCP questions pose later, or start with the (as I understand it) less technical FOCP for a gentler learning curve? submitted by /u/BLWHpurple to r/FinOps [link] [comments]

https://preview.redd.it/4ceym8fdealh1.png?width=1672&format=png&auto=webp&s=839cc0b352c7b923cacd02b4386bc834f48dc47d A lot of compliance work isn't spent fixing security problems. It's spent finding the evidence . Screenshots. Spreadsheets. Configuration exports. Change logs. Emails between IT, security, risk, and compliance teams. Then the cycle starts again before the next audit. For Saudi enterprises working across frameworks such as NCA ECC, SAMA, PDPL, ISO 27001, PCI DSS, SOC 2, and others, the operational burden can become enormous. We broke down a five-step approach to making compliance more continuous: Map infrastructure to relevant frameworks continuously Generate evidence automatically Detect configuration drift early Unify visibility across multi-vendor environments Use AI for policy and gap analysis—not just reporting The interesting shift is from: "We need to prepare for the audit." to: "We should already be able to prove our compliance posture." That's where AI-driven continuous monitoring could have a significant impact. Full breakdown: How to Cut Compliance Audit Effort by 90%: A Practical Framework for Saudi Enterprises For anyone working in security or compliance: what currently consumes the most time during your audit preparation? #Compliance #Cybersecurity #NCAECC #SaudiArabia #AgenticAI #GRC submitted by /u/WancloudsInc to r/FinOps [link] [comments]

We run an internal self-service platform API. Individual humans wanting CLI/scripted access this is where I'm stuck . Currently we mint a static token, show it once in the UI, and the user keeps it. Two things bother me: Authority is frozen at creation. We store the list of accounts the token may touch. If the user later loses their admin role on one of those accounts, the token keeps working. The credential outlives the entitlement. Distribution is copy-paste. It ends up in .env files, shell history, occasionally a chat message. If you've done per-request authorization lookups, what did it cost you in latency and directory load? How long do you cache, and how do you handle the lookup failing fail open or fail closed? For humans needing programmatic access, has anyone made short-lived tokens work by exchanging an existing SSO session? Feels like the "right" answer but I haven't seen it described much outside cloud provider SDKs. Is there a simpler option I'm missing? Something like mTLS with per-user certs, or just accepting static tokens with a short expiry and good auditing? For anyone who went the secret-manager route: did rotation actually work invisibly, or did you get outages from clients that cached the value? submitted by /u/SmartWeb2711 to r/devops [link] [comments]

For transparency: I run a small UK hosting company, where WHMCS is our current billing/provisioning system. Last week I moved WHMCS 8.13.1 from a CloudLinux OS 9 box running Plesk, to a fresh Debian 13 box. Mostly as we had WHMCS on a shared host during our early days and now it needed its own space. Two things worth sharing if anyone else hits this: Debian 13 ships with PHP 8.4 by default whereas WHMCS wants 8.3 so you need to use the Ondrej Sury repository and not the base repository. ionCube loader has to be placed manually and the ini file renamed to 00-ioncube.ini rather than the default 20-ioncube.ini otherwise if it loads after OPcache, WHMCS breaks very unhelpfully with no clear errors pointing at the fault. Also worth flagging, we had stale absolute paths from the old server that lived in multiple places. We had some in configuration.php, the storage settings gui, and the tblconfiguration SQL table and also in config.php that took a while to track down. Also don't forget to redo the cron! Posting in the hopes it saves someone else an afternoon! submitted by /u/SortingYourHosting to r/sysadmin [link] [comments]

Overview of the GUI I wanted a real OpenShift lab on AWS to practise on, and I specifically wanted UPI — user-provisioned infrastructure — because it's the install method where nothing is hidden. You build the VPC, the load balancers and the DNS records, and the installer only takes over once that's all correctly in place. IPI hides most of that inside the installer binary; ROSA is managed. The problem with UPI is that doing it by hand is slow and easy to get wrong, and I couldn't find anything that automated the whole lifecycle for AWS. So I wrote one. **What it does** `cluster.yaml` is the only file you edit. Everything else — the Terraform variables, the Ansible variables, `install-config.yaml` — is generated from it. ``` ocplab deploy # terraform apply, wait for bootstrap, approve CSRs, finalize ocplab verify # ClusterVersion, node readiness, ClusterOperators ocplab cost # what's deployed right now, priced live ocplab repair # recreate a worker AWS reclaimed, approve its CSRs ocplab power off # graceful shutdown, as an alternative to destroying ocplab destroy # ordered teardown, including what the cluster made itself ``` Some things it handles that took me a while to get right: - **Version pinning.** One field pins the installer, the `oc` client and the RHCOS AMI together, cached locally. - **Spot instances** for compute and bootstrap, with a "minimal" profile that uses them. Not for the control plane — UPI has no ControlPlaneMachineSet, so a reclaimed master isn't replaced automatically. - **Cost visibility.** `ocplab cost` reads live AWS state and prices it, Spot at Spot rates, and it's power-state aware. - **A budget safety net.** AWS Budget with alerts, an automatic lockdown action, and a daily killswitch Lambda — managed outside Terraform so it survives a failed teardown. - **Optional capability trimming**, which cuts the container images pulled per deploy by about a third. - **Teardown that actually completes.** The ingress operator creates a load balancer, a security group and DNS records outside Terraform; the teardown removes them in the right order first. **The web UI** There's also a local browser UI (`ocplab web start`) if you'd rather not watch a 40-minute deploy in a terminal: a status dashboard, a `cluster.yaml` editor that validates before saving, and live output for every operation — including Terraform's resource-by-resource log and the installer's own log. It binds 127.0.0.1 only and runs the same commands as subprocesses, so there's no second implementation to keep in sync. **Scope** It's built for labs and PoCs, not production. Single AZ, single NAT gateway, no HA design beyond etcd's own quorum. It costs roughly $0.83–1.06/hour while it's up, so the intended flow is create, test, destroy the same day. Tested end to end against OpenShift 4.22.6 and 4.22.7. MIT licensed. https://github.com/LuixyToledo97/openshift-upi-aws Happy to answer questions, and issues are welcome if you try it and something doesn't work. submitted by /u/Financial_Spring636 to r/devops [link] [comments]

I keep seeing engineers falling into an AI architecture trap. Some engineers try to solve every problem by prompting to a LLM. Others talk about creating an agent for every usecase. So knowing how to use AI has become a core competency. Otherwise you will waste time, money and DevOps resources. If you want to solve an actual business problem without creating an infrastructure nightmare, 99% of your use cases fall into one of these 5 application architectures. Here is how to know which one to use. 1. Basic Prompting (Stateless API) The Use Case: You need to summarize text, extract sentiment, or reformat data (like turning unstructured text into JSON). The data easily fits into the context window. The Trap: Thinking you need a custom model just because the base model didn't output your exact format on the first try. Before you do anything complex, try giving the model 3 concrete examples in the prompt (few-shot prompting). That solves the problem most of the time. 2. RAG (Retrieval-Augmented Generation) The Use Case: You need the LLM to answer questions about your proprietary data (internal wikis, codebase, customer records). You query a VectorDB for the relevant context, shove that context into the prompt, and let the LLM read it. The Trap: Trying to fine-tune a model on your company wiki. Models hallucinate facts. If you need accurate, source-traceable answers based on your internal data, you need RAG, not a fine-tuned model. 3. Chaining / Workflows The Use Case: The task is too complex for one prompt and requires distinct, predictable steps. For example: Step 1 extracts raw text -> Step 2 categorizes it -> Step 3 writes a summary based on the category. The Trap: Cramming a massive, 15-part instruction into a single mega-prompt and crossing your fingers. Break it down into a predictable pipeline. It’s much easier to debug a chain of small LLM calls than one giant, confused prompt. 4. Routing The Use Case: You are dealing with high volume and need to control costs and latency. You build a router that sends simple tasks (like spelling correction) to a fast, cheap model (like Claude Haiku or GPT-4o-mini), and complex reasoning tasks to the expensive, heavy models. The Trap: Sending every single trivial API request to the most expensive flagship model available and bankrupting your cloud account. 5. Autonomous Agents The Use Case: Open-ended tasks where the exact steps aren't known in advance. The LLM is given a goal and external tools (Python interpreter, web search, internal APIs) and loops continuously, evaluating its own output until it achieves the goal. The Trap: Using agents for deterministic workflows. If you can solve a problem with a standard if/then Python script or an Ansible playbook, do it. Do not introduce probabilistic, infinite-looping agents into a workflow that just needs a standard script. The Takeaway: Don't jump straight to training models. Work your way up this list. Start with prompting. If it lacks context, use RAG. If it lacks reliability, chain it. If it lacks cost-efficiency, route it. Only use agents when absolutely necessary. Curious if anyone else has horror stories of teams trying to fine-tune models to learn internal docs? submitted by /u/SeaworthinessHour233 to r/devops [link] [comments]

Claude Code, Gemini, Cloud Run This is a warning post. If you are about to build something on Google — Cloud, Gemini, any of it — read this first and then decide. I picked Google Cloud for my AI project because it looked like the serious choice. Gemini, Vertex AI, Cloud Run, all in one place. Solo dev, portrait retouching service, my own money, my own card, my own quota. Step one of the famous convenience: learning that everything in GCP is denied by default until you find the one permission buried nine menus deep among a million parameters with nearly identical names, and switch it on. Then the next one. Then the next one. Fine — that's the tax for a serious platform. I paid it. I ended up with nine Cloud Run services in asia-south1 and a pipeline I'm genuinely proud of. Then I went to work from the Himalayas, because I think better there. I was doing prompt research — controlled batches of 20-30 generations, reviewed by hand between batches, through Codex, Claude Code and Google's own Antigravity. Precisely the workflow Google advertises on stage. Google's security system looked at all of that and decided, without asking me, that it knew how to protect MY account better than I do. The one I pay for, with my own money. Banned. No reason. No warning. No policy cited. No email. Nothing. Convenience highlight #1: the services kept running. I checked — orchestrator returns 200, all eight processors return 405 to a GET, which is correct because they only accept POST. A stopped or deleted Cloud Run service returns 404. So they are alive, serving traffic, and billing me. Google did not decide my workload was dangerous. It decided it didn't want me near it. The robots stayed. The human was removed. Very cyberpunk. Convenience highlight #2: I was the sole Owner, and GCP never once suggested I add a backup. So there is now nobody on Earth who can administer this project, including Google's paying customer. Can't deploy, can't read logs, can't even shut it down to stop the charges. Convenience highlight #3: I sign in to Claude with that Google account. About SGD 150 a month. Also gone, also still charging. Anthropic can't help me either, because their recovery flow emails a code to the inbox Google just made unreadable. Google's security has now successfully protected me from a company that isn't Google, using money that isn't Google's. Now the part that made me write this. Two days ago Google emailed me: "Appeal approved." They reviewed it, they agreed, the account is restored, please sign in as soon as possible. I have not managed to sign in once since. "Too many failed attempts. You can continue in a few hours." Come back in a few hours. Prove you're not a robot. Prove it again. And again. A new obstacle every time, never a step forward, never one word about what I am supposed to do differently. I am not failing a security check — I cannot get far enough to take one. So I am solving captchas to prove I am human, for a system that flagged me for looking too automated, to reach an account that has officially already been given back to me. My appeal succeeded and nothing changed. An approved appeal that doesn't restore access isn't a resolution — it is written confirmation that they were wrong, followed by absolutely nothing. And here is the actual warning, which is bigger than my situation. Google has built the most user-hostile security system I have ever encountered, and I want to be precise about why, because "their security is annoying" doesn't capture it. Every security system trades off keeping the wrong people out against letting the right person in. Google has resolved that trade-off, permanently and in every product I have touched, in favour of assuming the account holder is the threat. It does not ask you. It does not warn you. It does not tell you afterwards. It decides — and then it makes you prove, over and over, to an automated gate with no human behind it, that you are the person whose name is on the card that pays for it. The part that should actually worry you: this is not the free consumer tier. This is the paid, professional, run-your-business-on-it tier. The same paternalism that locks somebody out of their family photos is sitting in front of production infrastructure with a credit card attached to it, and it is just as willing to fire. It knows better than you. It always knows better than you. In a photo app that's irritating. In front of your company's infrastructure it is a business risk that nobody puts on the pricing page. And because Google made one account the key to everything — mail, Cloud, identity for third-party services — one automated misfire doesn't take one thing from you. It takes the infrastructure, the mailbox, the ability to receive the recovery codes for everything else, and your access to services you buy from other companies. There is no blast door anywhere in that design. One decision, made by nobody, removes all of it at once. For the record, their own Terms of Service section 4.1 says that if Google believes you violated the AUP, Google will notify you and give you 24 hours to correct it before suspending. I received zero notifications and zero hours. Section 4.2 permits instant suspension to "protect the Services" — and the Services, as established above, are magnificently unprotected and still running. What I would actually tell you to do: If you're on GCP as a solo dev, add a second Owner account to every project. Today. I can't anymore, and that is the whole lesson. Don't let one Google account be your identity for anything you can't afford to lose. Use a real email you control for third-party sign-ins. Keep your infrastructure somewhere you can still reach when Google decides, without telling you, that you are suspicious. So here is where this has actually left me. Instead of moving my product forward — the thing I was doing, with my own money, on their platform, in the weeks I had set aside for it — I spend my days solving Google's captchas and writing Google letters. Not to build anything. Not even to get back into Google's own product. To regain access to a service that is not Google's, that I pay a different company for, and that Google is still, right now, blocking me from reaching. That is what their security achieved. Not one bad actor stopped. One paying developer taken off his own work for days, doing unpaid administrative labour to undo an error nobody will explain, while the meter runs. I am not saying the products are bad. Gemini is very good, and I built here on purpose. I am saying that the company running them will take them away from you without a word, hand you an appeal form, approve the appeal, and still leave you outside the door — and that you should price that in before you build a business on it. Extremely convenient platform. Would get banned again. ^(This was written with the help of an AI so I don't shock anyone with my natural vocabulary. Peace and love to everybody — including, reluctantly, Google.) submitted by /u/Key_Lengthiness1518 to r/googlecloud [link] [comments]

I'm an indie dev and honestly exhausted. I've been stuck in a support loop with Google Cloud for nearly a full year over an acknowledged platform-side bug. Here is the short version of what's happening: The Bug: My GCP project was hit with an unexpected $8,367.11 charge surge on the Gemini 2.5 Flash image generation SKU. Product Lead Logan Kilpatrick publicly acknowledged this bug on the official forums and promised affected devs would be taken care of. The "90% Cap": Google Billing gave a 90% adjustment and declared that 90% is the "maximum allowable limit" for this incident—leaving $838.26 on my account for a bug on Google's infrastructure. The "No Logs" Excuse: When I demanded backend logs or proof showing why I owe this remaining balance, support literally replied: "they do not share external-facing documentation, internal case notes, or further detailed explanations regarding their final review process." 1 Year of Support Ping-Pong: Opened a billing ticket $\rightarrow$ immediately closed and bounced back. Tier 1 agents just rotate shifts, paste identical templates, and close tickets without consent. My account currently sits with a red banner warning and an active $838.26 balance for services I never consumed. Screenshots attached: 1.Billing Support confirming the charge was caused by the Gemini 2.5 Flash bug. https://preview.redd.it/x083tf255olh1.png?width=438&format=png&auto=webp&s=f57d82c545e94f41d28eee18802a0243731f2366 Support stating the 90% adjustment is final. https://preview.redd.it/eaq8leu55olh1.png?width=690&format=png&auto=webp&s=36fb40b01c9f27c07cbd0f1d920eacd011b0575d My billing console showing the pending $838.26 balance and account warning. https://preview.redd.it/ahyl62n65olh1.png?width=690&format=png&auto=webp&s=e9403517ca4b0c93b3fbcc102633f78f35cf7f4b Support's latest response admitting they cannot provide any logs or explanation. https://preview.redd.it/ec9uxzv95olh1.png?width=1481&format=png&auto=webp&s=a59b05554a4fd661828dc2d83d32d9a7178ad05a I've documented this on the Google AI Forum and am preparing consumer complaints in Japan (消費者庁 / 消費生活センター). Has anyone ever broken through this Tier 1 "final decision" wall without an Enterprise support plan? Any advice or visibility would be really appreciated. submitted by /u/Character-Candy1120 to r/googlecloud [link] [comments]

This is likely a bit of a niche question, but I am going a tad insane trying to figure out why my old script for purging emails via a compliance search is failing. Previously (around a year ago), this script would create a new compliance search in EXO, start it, poll for progress, and then export the results and ask if you wanted to purge the emails from the relevant mailboxes. Had something come up yesterday while I was unavailable, and my old script didn't work. Messing around with it this morning, I found that while I can create a new compliance search, I cannot preview the results OR export the results. It does however, seem like I can still purge emails. Only you know, I cannot preview the emails, or verify it actually found the right emails beyond hoping that the item count from the search makes sense. Not so cool. From what I found on Microsoft's own documentation pages, it looks like the preview, and export functions will only work for on-prem exchange now, and I really do not see any alternatives. All that being said, is this Microslop making things difficult for the sake of adding some new subscription in the coming months, or is there actually a way to do this and their documentation is woefully out of date? I have a feeling the answer is going to be a resounding "no" based on an hour or so of searching, but you never know 😉 submitted by /u/TheOnlyKirb to r/sysadmin [link] [comments]

Hi guys, I'm needing your guidance here. I work in an automation-focused role that gradually became much more technical than operational. A lot of my job is taking broken/manual processes, improving them, and building automations around them. In practice, that means dealing with business rules, integrations, permissions, data sources, testing, edge cases, etc. However, I still consider myself junior technically. My current challenge is that the company is becoming more restrictive about external/unapproved automation tools, so I’m trying to move everything into approved internal tools and infrastructure. The problem is that access is granted gradually, permissions change, and some features depend on other teams. Stakeholders often just see that “the automation isn’t ready,” while a lot of the delay is actually caused by access, security, infrastructure, or dependencies. I’ve started communicating blockers and development stages more clearly, but I’m curious about how this is normally handled in US/European companies. How much responsibility is usually placed on the developer when delivery is blocked by access or another team? And how do you manage expectations when something sounds simple from the business side but isn’t simple technically? submitted by /u/Kooky-Internet8806 to r/devops [link] [comments]

https://preview.redd.it/tj068f1g1cjh1.png?width=2107&format=png&auto=webp&s=35fb64e37f0fe01bc985df9e0407adee6f055cee https://preview.redd.it/7a3lvq2k1cjh1.png?width=1695&format=png&auto=webp&s=da48778e9b362485b942eb4f05a59f1b7256cb40 My usage this month was around IDR 170.000 but I was charged IDR 3 million. Why is that? The support bot said: Since your July usage ( Rp 863,006 ) was already paid and your August usage is only ( Rp 168,763 ), these charges of IDR 1,000,000 and IDR 2,000,000 processed today are highly likely due to one of the following reasons: Temporary Authorization Holds: When you update payment details, add a new card, or reach certain verification checkpoints, Google or your bank may place temporary authorization holds (often in round numbers like IDR 1,000,000 or 2,000,000) to verify your card's validity. These are not actual charges and will be automatically released back to your account by your bank within a few business days. Multiple Google Billing Accounts: If you use the same credit card for other Google services (such as Google Ads, Google Workspace, or a separate Google Cloud Billing account), these charges may have originated from those services. submitted by /u/rfajr to r/googlecloud [link] [comments]

We got the Azure notice about upgrading our Storage and Blob accounts from GPv1 to GPv2 ahead of the October 13th automatic migration deadline. The Azure portal makes it look incredibly easy with a simple "one-click" in-place upgrade button for the Storage Accounts which have been identified as GPv1 and Blob. Documentation says zero downtime and zero data loss. However, I know GPv2 flips the billing model (cheaper storage, much higher transaction costs). For anyone who has already gone through this migration: Did you just click the upgrade button and wing it? If so, did your bill spike unpredictably? Did you actually pull Azure Monitor metrics first? If you audited transaction volumes, what thresholds made you hesitate or re-architect a workload? Any hidden gotchas? Did you run into issues with default access tiers (Hot vs. Cool) or legacy ZRS replication during the flip? Since Microsoft is going to auto-upgrade us anyway in October, I want to know if it is worth digging into the transaction logs manually or if I am overthinking a routine upgrade. Appreciate any real-world horror stories or "it went fine" reassurances! submitted by /u/nomadicviking024 to r/AZURE [link] [comments]

submitted by /u/codingdecently to r/FinOps [link] [comments]

After working with Kubernetes in production, I've noticed that some of the most annoying incidents aren't caused by obvious failures. They're often caused by small configuration decisions that look perfectly reasonable during review. Things like: missing resource requests/limits incorrect probes overly permissive RBAC missing PodDisruptionBudgets unsafe container configuration incorrect readiness behaviour services without appropriate timeouts configuration drift between environments I'm curious what the DevOps community has actually encountered in production. What's one Kubernetes configuration mistake that caused you a real incident? I'd especially like to hear about the less obvious ones that aren't caught by the usual linters. submitted by /u/nerd3n to r/devops [link] [comments]

What happened On Aug 1, an IAM access key belonging to one of our accounts was compromised through a supply chain compromise — the key was exposed via a third party in our toolchain, not by us publishing it. This was a programmatic access key, so MFA (enforced on all human identities from day one) was never in the path. We caught it and reported it to AWS Support within 4 hours , well inside the 48-hour public-contract cancellation window. The case then sat unassigned for 14 days. As of today that's 20 days, still open, still no resolution. Where the bill is now Total USD 477,505.20 Contract creation (0 months) USD 4,800.00 Usage Fee | us-east-1 USD 472,705.20 47,270,520 Units @ USD 0.01 The part I don't understand We never activated the product. Specifically: - The Marketplace console still shows "Set up product" for this agreement - No License ID was ever issued - CloudTrail shows zero Marketplace events in our account after the initial compromise window on Aug 1 - We have never had credentials for, or logged into, anything on the seller's side Yet 47.2 million billable units were metered against us in us-east-1. As I understand the SaaS flow, usage on this kind of product is submitted by t he seller calling `BatchMeterUsage` against the entitlement's `CustomerIdentifier` — from the seller's own infrastructure, not from our account. If that's right, then nothing in our account was ever in the path, and no control we have could have stopped it. Revoking the key, deleting the attacker's IAM user, applying SCPs — none of it touches seller-side metering. The unit count has been static for several days now, so metering appears to have stopped, but the agreement status is the thing I can't get a straight answer on. Support so far Seller support (automated) told us Marketplace transactions are not their department and to contact AWS. AWS support has pointed at the seller for anything usage-related. The payer account has now escalated and asked AWS to investigate the agreement and stop metering. Still waiting. There is no phone line and no chat. The only channel is tickets, and they've been sitting for 20 days. Questions for people who actually know the internals For a SaaS contract-with-consumption product, is metering purely seller-side ? Is there any circumstance where usage gets attributed to a buyer account without the buyer ever completing registration? Does "Set up product" persisting in the console reliably mean the fulfillment/`ResolveCustomer` handshake never happened — or is that just stale console state that doesn't update? Is there any way for a buyer to see the registration record for their own entitlement? Anything in the Agreement APIs, CUR, or elsewhere that shows when/whether `ResolveCustomer` was called? The 48-hour window: AWS Customer Service can process a full refund on a public contract without seller involvement. Does that path survive a case sitting unassigned for 14 days, or is the window enforced strictly on wall-clock time regardless of AWS-side latency? Has anyone here had Marketplace usage fees (not just the contract fee) reversed after a credential compromise? Contract fee reversals I've seen written up. Usage fees at this scale, never. To be blunt about the stakes: we're a small company. We cannot pay $472k for consumption we did not generate, on a product we never activated, after cutting off access in under 4 hours. I'm trying to understand the metering mechanics, and whether there's a path here other than waiting on a ticket queue. submitted by /u/Round_Vegetable3764 to r/aws [link] [comments]