I Followed the Official AWS Amplify Guide and Was Charged $1,100

Hard spend limits are an anti-feature for enterprise customers, who are the core customer of AWS. Almost no level of accidental spend is worth creating downtime or data loss in a critical application.

Even having the option of a hard spend limit would be hazardous, because accounting teams might push the use of such tools, and thereby risk data loss incidents when problems happen.

Hard spend limits might make sense for indie / SME focused cloud vendors though.

But the presence of a feature does not mean it has to be used?

Cool, have an "I know what I'm doing, opt me out!" button.

> Even having the option of a hard spend limit would be hazardous, because accounting teams might push the use of such tools, and thereby risk data loss incidents when problems happen.

"Hazardous" feels like the wrong word here - if your customer decides to enact a spend limit it should not be up to you to decide whether that's good for them or not.

> Hard spend limits are an anti-feature for enterprise customers, who are the core customer of AWS. Almost no level of accidental spend is worth creating downtime or data loss in a critical application.

Great, so they don’t have to use the feature?

That excuse was a great excuse when AWS was an MVP for someone. 20+ years later… there is no excuse.

The accounting teams of many orgs would want this feature to be enabled, but the tech teams wouldn't. Asking AWS not to add this feature means the tech teams win the debate before it starts.

So have a “Starter” account with spend limits then, don’t understand how an individual is supposed to learn this stack and actually sleep at night without waking up panicking something has been left running.

> Almost no level of accidental spend is worth creating downtime or data loss in a critical application.

That feels like a bit of a red herring — if that was their ethos, then you'd _have_ to choose burstable/autoscaling config on every service. If I can configure a service to fall over rather than scale at a hard limit, that points to them understanding different their use cases (prod vs dev) and customer types (start-up vs enterprise).

Additionally, anytime I've worked for an enterprise customer, they've had a master service agreement set-up with AWS professional services rather than entering credit card info, so they could use that as a simple way to offer choices.

Is every problem you see so insurmountable?

In the olden days if we spotted a customer ringing up a colossal bill, we would tell them. These huge Amazon bills are fast but still multiple days. They can trivially use rolling-projection windows to know when an account is having a massive spike.

They could use this foresight to call the customer, ensure they're informed, give them the choice about how to continue. This isn't atomic rocket surgery.

"Oh but profit" isn't an argument. They are thousands of dollars up before a problem occurs. The only money they lose is money gained through customer accident. Much of it forgiven to customers who cannot afford it. It's not happily spent. They can do better business.

By checking what resources you are spinning up and double checking everything has been removed after you're done. I have used AWS for small projects many times and have never been hit with a surprise bill. The platform is built for actual customers, not your hello world app.

Being aware of everything you're running on AWS is trivial when it's all on the EC2 dashboard, and you create everything by hand or using Terraform.

Many of these new AWS-provided stacks, however, seem to create stuff all over your account.

The moral of the story? Don't ever use AWS tools like the one the OP describes, ones which create a bunch of different resources for you automatically.

This guy is a straight shooter with upper management written all over him.

(Office Space)

So make it an option. That’s not hard.

Make "unlimited spend" an opt-in. That way users who have explicitly chosen this and agreed to the terms can't then complain to support to try and get the bill waived.

AWS provides forecasted spend and other alerts. People just don't use them and then complain.

> Even having the option of a hard spend limit would be hazardous, because accounting teams might push the use of such tools

Tell me you're Shadow IT without telling me you're Shadow IT.

I know legitimizing shadow IT is still the value proposition of AWS to a lot of organizations. But it sucks if that's the reason the rest of us can't get an optional feature.

That seems like a very hypothetical problem you are solving there..

> Even having the option of a hard spend limit would be hazardous, because accounting teams might push the use of such tools, and thereby risk data loss incidents when problems happen.

This is an interesting point and something I can totally imagine happening! I guess if you have fixed spending limits in a large enough organisation, you lose some of the benefit of cloud infrastructure. Convincing a (metaphorically) remote Finance department to increase your fixed spending limit is probably a tougher task than ordering a load of new hardware!

> Hard spend limits are an anti-feature for enterprise customers

Yada yada yada, that's the same old excuse the cloud providers trot out.

Now, forgive me for my clearly Nobel Prize winning levels of intellect when I point out the following...

Number one: You would not have to turn on the hard spend limit if such functionality were to be provided.

Number two: You could enable customers to set up hard limits IN CONJUNCTION WITH alerts and soft limits, i.e. hitting the hard limit would be the last resort. A bit like trains hitting the buffers at a station ... that is preferable to killing people at the end of the platform. The same with hard spend limits, hitting the limit is better than waking up in the morning to a $1m cloud bill.

> The moral of the story? Don't ever use AWS tools like the one the OP describes, ones which create a bunch of different resources for you automatically.

i use them to create a small test stack to look at it for a day or two.

then go through, delete all of the resources, put what i need into terraform etc.

has worked well for me in the past.

but yeah, i would never blindly use aws tools to magically put something into production.

Cost Anomaly alerts do exactly this.

You can't even setup Cloudwatch alerts on your actual spend, the only metric available to you is "EstimatedCharges".

> Almost no level of accidental spend is worth creating downtime or data loss in a critical application

But not all applications are critical, and the company deploying those applications should be able to differentiate between what's critical and what's not. If they're unable to, that's their fault. If there's no option to set hard limits, that's AWS' fault.

If a company chooses to be run by accounting rather than by tech, that's fine; they should get the good and the bad outcomes that come from that choice.

Which is great if you understand them and set them up. The post here is an experience following an Amazon tutorial and without expecting it, ending up $$$ in debt.

Again, Amazon isn't stepping up to protect users from themselves. They could do a lot more.

There's no way to implement a hard limit without getting in the middle of your system in ways that (a) alter the system design, (b) in ways you cannot correct for, and (c) not for the better.

A service doesn't just "fall over" at a limit. There has to be other machinery to limit it.

Given that all your usage and traffic other than that at the limit request should not be gated or limited, why would you want someone else injecting additional complexity and bottleneck risk inline?

Determine your own graceful envelope, implement accordingly.

The OP was saying that having any spending limit would be antithetical to enterprise usage because it could bring critical services to a halt.

My point was, if that's a reason to have unbounded spending, why allow me to spin up a service that can get CPU or RAM bound?

> Determine your own graceful envelope, implement accordingly.

Which most people do have, but we then also want an "ungraceful" backstop — trains have both a brake and a dead man's switch.

They refunded him the money, without a lot of hassle. He toddled into a rough edge, bumped his head, Amazon made it better and apparently gave him a lollipop as well and sent him on his way. I think Amazon should get some credit here.

In my experience every AWS service I've worked with has an API to destroy the resource, eg destroy RDS instance without backup, terminate EC2, etc. I want it to irrecoverably destroy everything in my account when it hits the limit, no questions asked.

Of course there is. If someone hits their spending limit, asynchronously shut off the services (using the same API call that your customers can use, so no need to alter the system).

Then apply the hard limit in the billing code. If it took a minute or two to shut off all the instances, maybe the customer's bill should have been $1.001M instead of $1M, but cap the bill to $1M anyway. Given their profit margins of x,000% I think they can afford the lost pennies.

I feel pain just reading this because I know where that guy is coming from; at many orgs as soon as some stupid thing is available it gets mandated by people who have no idea how it works or what impacts it will have, all the time. If it was just as simple as not enabling the stupid option surely we could just not ena...

Excuse me for a minute, I have to go reset my password that expires every 10 days and can not match any previous password and enter my enterprise mandated sms 2fa because authenticator scary -- woops my SharePoint session expired after 120 seconds of inactivity let me just redo that -- oh what's that my corporate vpn disconnected because I don't have the mandatory patches applied, let me just restart -- Woah would you look at that my network interface doesn't work anymore after that update -- yes yes I'm sorry I know my activity light on MS Teams turned yellow for 5 minutes I'm working on it, just gotta do these other 12 steps so I can reset my password to -- oh look it's time to fill out the monthly company values survey, what's that, it's due by end of day?

> There's no way to implement a hard limit without ...

I don't believe that for a minute.

You know why ?

Let's turn it on its head. What happens if the credit card on your AWS account expires or is frozen ?

You think AWS are going to continue letting you rack up money with no card to charge against ?

I betcha they'll freeze your AWS assets the nanosecond they can't get a charge against the card.

The mechanism for customer-defined hard limits is basically no different.

From experience they will let you spend tens of thousands of dollars a month without a valid payment method.

Like many times more than you've ever even spent with them.

I mean it's like 6 months of that before you even get your first non-standard form email.

Well, then enterprise won't use it.

OTOH for example the default quota of 55PB (yes, go check it) for your daily limit of data extraction from GBQ is funny, until you make a costly mistake, or some forked process turns zombie.

This is predatory practice, that I can't set up MONEY limits for cloud services.