←back to thread

How when AWS was down, we were not

(authress.io)
203 points mooreds | 4 comments | 17 Nov 25 17:07 UTC | HN request time: 0s | source
Show context
rdoherty ◴[17 Nov 25 21:02 UTC] No.45958246[source]▶
This is probably one of the best summarizations of the past 10 years of my career in SRE. Once your systems get complex enough, something is always broken and you have to prepare for that. Detection & response become just as critical as pre-deploy testing.

I do worry about all the automation being another failure point, along with the IaC stuff. That is all software too! How do you update that safely? It's turtles all the way down!

replies(2): >>45958417 #>>45958775 #
evanmoran ◴[17 Nov 25 21:16 UTC] No.45958417[source]▶
Iac is definitely a failure point, but the manual alternative is much worse! I’ve had a lot of benefit from using pulumi, simply because the code can be more compact than the terraform hcl was.

For example, for the fall over regions (from the article) you could make a pulumi function that parameterizes only the n things that are different per fall over env and guarantee / verify the scripts are nearly identical. Of course, many people use modules / terragrunt for similar reasons, but it ends up being quite powerful.

replies(3): >>45958669 #>>45958804 #>>45958816 #
xyzzy123 ◴[17 Nov 25 21:55 UTC] No.45958804[source]▶
I actually like terraform for its LACK of power (tho yeah these days when I have a choice I use a lot of small states and orchestrate with tg).

Pulumi or CDK are for sure more powerful (and great tools) but when I need to reach for them I also worry that the infra might be getting too complex.

replies(2): >>45958835 #>>45958977 #
1. yearolinuxdsktp ◴[17 Nov 25 22:14 UTC] No.45958977[source]▶
IMO Pulumi and CDK are an opportunity to simplify your infra by capturing what you’re working with using higher-level abstractions and by allowing you to refactor and extract reusable pieces at any level. You can drive infra definitions easily from typed data structures, you can add conditionals using natural language syntax, and stop trying to program in a configuration language (Terraform HCL with surprises like non-short-circuited AND evaluation).

You still end up having IaaC. You can still have a declarative infrastructure.

replies(3): >>45959275 #>>45959342 #>>45963695 #
2. xyzzy123 ◴[17 Nov 25 22:44 UTC] No.45959275[source]▶
Absolutely, the best case is it's much better, safer, readable etc. However, the worst case is also worse. From the perspective of someone who provides devops support to multiple teams, terraform is more "predictable".
3. andrewaylett ◴[17 Nov 25 22:52 UTC] No.45959342[source]▶
That's how we use CDK. Our CDK (in general) creates CloudFormation which we then deploy. As far as the tooling which we have for IaC is concerned, it's indistinguishable from hand-written CloudFormation — but we're able to declare our intent at a higher level of abstraction.
4. darkwater ◴[18 Nov 25 11:17 UTC] No.45963695[source]▶
> and stop trying to program in a configuration language

Many people don't program with a configuration language like HCL. We use it as what it is - a DSL - that covers its main use case in an elegant manner. Maybe I never touched complex enough infra that twists a DSL into a general-use language, but in my experience there are simply no real benefits when using something like CDK (I never tried Pulumi to be fair).