Most active commenters

    ←back to thread

    Ask HN: Our AWS account got compromised after their outage

    391 points kinj28 | 18 comments | 21 Oct 25 15:55 UTC | HN request time: 0.432s | source | bottom

    Could there be any link between the two events?

    Here is what happened:

    Some 600 instances were spawned within 3 hours before AWS flagged it off and sent us a health event. There were numerous domains verified and we could see SES quota increase request was made.

    We are still investigating the vulnerability at our end. our initial suspect list has 2 suspects. api key or console access where MFA wasn’t enabled.

    1. sousastep ◴[21 Oct 25 16:44 UTC] No.45657982[source]▶
    couple folks on reddit said while they were refreshing during the outage, they were briefly logged in as a whole different user
    replies(6): >>45658079 #>>45658884 #>>45659047 #>>45659106 #>>45659578 #>>45665172 #
    2. afandian ◴[21 Oct 25 16:51 UTC] No.45658079[source]▶
    Got references? This is crazy.
    replies(2): >>45658716 #>>45662414 #
    3. ◴[21 Oct 25 17:30 UTC] No.45658716[source]▶
    4. __turbobrew__ ◴[21 Oct 25 17:41 UTC] No.45658884[source]▶
    Maybe dynamodb was inconsistent for a period and as that backs IAM credentials were scrambled? Do you have references to this, because if it is true that is really really bad.
    replies(1): >>45669251 #
    5. CaptainOfCoit ◴[21 Oct 25 17:49 UTC] No.45659047[source]▶
    > couple folks on reddit said while they were refreshing during the outage, they were briefly logged in as a whole different user

    Didn't ChatGPT have a similar issue recently? Would sound awfully similar.

    replies(1): >>45660022 #
    6. liviux ◴[21 Oct 25 17:53 UTC] No.45659106[source]▶
    A friend of a friend knows a friend who logged in to Netflix root account. Source: trust me bro
    7. gwbas1c ◴[21 Oct 25 18:23 UTC] No.45659578[source]▶
    Years ago I worked for a company where customers started seeing other customers' data.

    The cause was a bad hire decided to do a live debugging session in the production environment. (I stress bad hire because after I interviewed them, my feedback was that we shouldn't hire them.)

    It was kind of a mess to track down and clean up, too.

    8. sunaookami ◴[21 Oct 25 18:55 UTC] No.45660022[source]▶
    Steam also had this, classic caching issue.
    replies(1): >>45661165 #
    9. mbo ◴[21 Oct 25 20:20 UTC] No.45661165{3}[source]▶
    This happened to me on Twitter maybe like, 9 years ago? What's the mechanism of action that causes this to happen?
    replies(1): >>45662696 #
    10. blast ◴[21 Oct 25 22:16 UTC] No.45662414[source]▶
    I saw a link to https://old.reddit.com/r/webdev/comments/1obtbmg/aws_site_re... at one point but then it was deleted
    replies(3): >>45662821 #>>45662923 #>>45666933 #
    11. howinator ◴[21 Oct 25 22:47 UTC] No.45662696{4}[source]▶
    The easiest way to do this is to misconfigure your CDN so that it caches set-cookie headers.
    12. duk3luk3 ◴[21 Oct 25 22:59 UTC] No.45662821{3}[source]▶
    This isn't about an aws account, this is about the auth inside the project that user is running.
    13. perpil ◴[21 Oct 25 23:09 UTC] No.45662923{3}[source]▶
    This is not about the AWS Console. It is talking about the customer's site hosted on CloudFront. It is possible to cross wires with user sessions when using CloudFront if you haven't set caching granular enough to be specific to an end user. This scenario is customer error, not AWS.
    replies(1): >>45665707 #
    14. TZubiri ◴[22 Oct 25 05:25 UTC] No.45665172[source]▶
    A security incident like this would dwarf in comparision to partial unavailability of services.
    15. fulafel ◴[22 Oct 25 06:56 UTC] No.45665707{4}[source]▶
    I'd argue it's a classic footgun and a flaw of CloudFront (they should at least warn about it much more).
    16. CodesInChaos ◴[22 Oct 25 10:05 UTC] No.45666933{3}[source]▶
    electricity_is_life's comment on reddit seems to explain it:

    > Not sure if this is what happened to you, but one thing I ran into a while back is that even if you return Cache-Control: no-store it's still possible for a response to be reused by CloudFront. This is because of something called a "collapse hit" where two requests that occur at the same time and are identical (according to your cache key) get merged together into a single origin request. CloudFront isn't "storing" anything, but the effect is still that a user gets a copy of a response that was already returned to a different user.

    > https://stackoverflow.com/a/69455222

    > If your app authenticates based on cookies or some other header, and that header isn't part of the cache key, it's possible for one user to get a response intended for a different user. To fix it you have to make sure any headers that affect the server response are in the cache key, even if the server always returns no-store.

    ---

    Though the AWS docs seem to imply that no-store is effective:

    > If you want to prevent request collapsing for specific objects, you can set the minimum TTL for the cache behavior to 0 and configure the origin to send Cache-Control: private, Cache-Control: no-store, Cache-Control: no-cache, Cache-Control: max-age=0, or Cache-Control: s-maxage=0.

    https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...

    replies(1): >>45668186 #
    17. phyzome ◴[22 Oct 25 12:41 UTC] No.45668186{4}[source]▶
    Collapse-hits... hadn't thought about those in years. Brought back some trauma.
    18. aeyes ◴[22 Oct 25 13:57 UTC] No.45669251[source]▶
    AWS IAM doesn't use or depend on DynamoDB