←back to thread

391 points kinj28 | 2 comments | | HN request time: 0.001s | source

Could there be any link between the two events?

Here is what happened:

Some 600 instances were spawned within 3 hours before AWS flagged it off and sent us a health event. There were numerous domains verified and we could see SES quota increase request was made.

We are still investigating the vulnerability at our end. our initial suspect list has 2 suspects. api key or console access where MFA wasn’t enabled.

Show context
sousastep ◴[] No.45657982[source]
couple folks on reddit said while they were refreshing during the outage, they were briefly logged in as a whole different user
replies(6): >>45658079 #>>45658884 #>>45659047 #>>45659106 #>>45659578 #>>45665172 #
afandian ◴[] No.45658079[source]
Got references? This is crazy.
replies(2): >>45658716 #>>45662414 #
blast ◴[] No.45662414[source]
I saw a link to https://old.reddit.com/r/webdev/comments/1obtbmg/aws_site_re... at one point but then it was deleted
replies(3): >>45662821 #>>45662923 #>>45666933 #
1. CodesInChaos ◴[] No.45666933[source]
electricity_is_life's comment on reddit seems to explain it:

> Not sure if this is what happened to you, but one thing I ran into a while back is that even if you return Cache-Control: no-store it's still possible for a response to be reused by CloudFront. This is because of something called a "collapse hit" where two requests that occur at the same time and are identical (according to your cache key) get merged together into a single origin request. CloudFront isn't "storing" anything, but the effect is still that a user gets a copy of a response that was already returned to a different user.

> https://stackoverflow.com/a/69455222

> If your app authenticates based on cookies or some other header, and that header isn't part of the cache key, it's possible for one user to get a response intended for a different user. To fix it you have to make sure any headers that affect the server response are in the cache key, even if the server always returns no-store.

---

Though the AWS docs seem to imply that no-store is effective:

> If you want to prevent request collapsing for specific objects, you can set the minimum TTL for the cache behavior to 0 and configure the origin to send Cache-Control: private, Cache-Control: no-store, Cache-Control: no-cache, Cache-Control: max-age=0, or Cache-Control: s-maxage=0.

https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...

replies(1): >>45668186 #
2. phyzome ◴[] No.45668186[source]
Collapse-hits... hadn't thought about those in years. Brought back some trauma.