←back to thread

ByteDance sacks intern for sabotaging AI project

(www.bbc.com)
192 points beedeebeedee | 1 comments | 21 Oct 24 03:21 UTC | HN request time: 0s | source
Show context
peterkos ◴[21 Oct 24 04:04 UTC] No.41900587[source]▶
I'm reminded of a time that an intern took down us-east1 on AWS, by modifying a configuration file they shouldn't have had access to. Amazon (somehow) did the correct thing and didn't fire them -- instead, they used the experience to fix the security hole. It was a file they shouldn't have had access to in the first place.

If the intern "had no experience with the AI lab", is it the right thing to do to fire them, instead of admitting that there is a security/access fault internally? Can other employees (intentionally, or unintentionally) cause that same amount of "damage"?

replies(12): >>41900622 #>>41900627 #>>41900641 #>>41900805 #>>41900919 #>>41901069 #>>41901814 #>>41903916 #>>41909887 #>>41910021 #>>41910134 #>>41910235 #
grogenaut ◴[21 Oct 24 04:15 UTC] No.41900641[source]▶
From what I've seen in Amazon it's pretty consistent that they do not blame the messenger which is what they consider the person who messed up. Usually that person is the last in a long series of decisions that could have prevented the issue, and thus why blame them. That is unless the person is a) acting with malice, b) is repeatedly shown a pattern of willful ignorance. IIRC, when one person took down S3 with a manual command overriding the safeguards the action was not to fire them but to figure out why it was still a manual process without sign off. Say what you will about Amazon culture, the ability to make mistakes or call them out is pretty consistently protected.
replies(4): >>41900811 #>>41901212 #>>41911207 #>>41914419 #
tgavVs ◴[21 Oct 24 04:56 UTC] No.41900811[source]▶
> From what I've seen in Amazon it's pretty consistent that they do not blame the messenger which is what they consider the person who messed up

Interesting that my experience has been the exact opposite.

Whenever I’ve participated in COE discussions (incident analysis), questions have been focused on highlighting who made the mistake or who didn’t take the right precautions.

replies(5): >>41900843 #>>41900913 #>>41901176 #>>41901751 #>>41902023 #
Cthulhu_ ◴[21 Oct 24 08:51 UTC] No.41902023{3}[source]▶
But when that person was identified, were they personally held responsible, bollocked, and reprimanded or were they involved in preventing the issue from happening again?

"No blame, but no mercy" is one of these adages; while you shouldn't blame individuals for something that is an organization-wide problem, you also shouldn't hold back in preventing it from happening again.

replies(1): >>41907147 #
1. grogenaut ◴[21 Oct 24 18:50 UTC] No.41907147{4}[source]▶
Usually helping prevent the issue, training. Almost everyone I've ever seen cause an outage is so "oh shit oh shit oh shit" that a reprimand is worthless, I've spent more time a) talking them through what they could have done better and, encouraging them to escalate quicker b) assusaging their fears that it was all their fault and they'll be blamed / fired. "I just want you to know we don't consider this your fault. It was not your fault. Many many people made poor risk tradeoffs for us to get to the point where you making X trivial change caused the internet to go down"

In some cases like interns we probably just took their commit access away or blocked their direct push access. Now a days interns can't touch critical systems and can't push code directly to prod packages.