←back to thread

192 points beedeebeedee | 3 comments | | HN request time: 0s | source
Show context
peterkos ◴[] No.41900587[source]
I'm reminded of a time that an intern took down us-east1 on AWS, by modifying a configuration file they shouldn't have had access to. Amazon (somehow) did the correct thing and didn't fire them -- instead, they used the experience to fix the security hole. It was a file they shouldn't have had access to in the first place.

If the intern "had no experience with the AI lab", is it the right thing to do to fire them, instead of admitting that there is a security/access fault internally? Can other employees (intentionally, or unintentionally) cause that same amount of "damage"?

replies(12): >>41900622 #>>41900627 #>>41900641 #>>41900805 #>>41900919 #>>41901069 #>>41901814 #>>41903916 #>>41909887 #>>41910021 #>>41910134 #>>41910235 #
donavanm ◴[] No.41910021[source]
I worked at AWS for 13 years. I did “aws call leader” for 7 years, and worked in the reliability org when we rebuilt the coe tool. Ive personally blown up a service or two, and know other PEs whove done the same or larger.

Ive never heard of an individual being terminated or meaningfully punished for making an earnest mistake, regardless of impact. I do know of people who were rapid term’d for malicious, or similar, actions like sharing internal information or (attempting to) subvert security controls.

On the whole I did see Amazon “do the right thing” around improving process and tools; people are a fallible _part_ of a system, accountability requires authority, incremental improvements today over a hypothetical tomorrow.

replies(1): >>41910958 #
1. zmgsabst ◴[] No.41910958[source]
PAM debacle (17Q4) in Device Econ is a counter example.

And that wasn’t even a mistake the SDEs made — they were punished for the economists being reckless and subsequently bullied out of the company, despite the SDEs trying to raise the alarm the whole time.

replies(1): >>41911006 #
2. donavanm ◴[] No.41911006[source]
Is that devices as in digital/alexa land? Never had too much overlap there. AWS and CDO were discrete for incident and problem management after ‘14 or soz
replies(1): >>41913294 #
3. zmgsabst ◴[] No.41913294[source]
Yeah — my point was Amazon is very large and standards vary. I won’t pretend I know the whole picture, but I’ve seen retaliation against SDEs multiple times.

I’ve heard mixed things about CDO, positive things about AWS, but where I worked in Devices and FinTech were both wild… to the point FinTech (circa 2020) didn’t even use the PRFAQ/6-pager methodology. Much to the surprise of people in CDO I asked for advice.