Summary of the Amazon DynamoDB Service Disruption in US-East-1 Region

(aws.amazon.com)
92 points meetpateltech | 9 comments | 23 Oct 25 01:19 UTC | HN request time: 0.74s | source | bottom
1. ◴[23 Oct 25 01:24 UTC] No.45677169[source]▶
2. ◴[23 Oct 25 01:41 UTC] No.45677270[source]▶
3. joeyhage ◴[23 Oct 25 01:57 UTC] No.45677360[source]▶
> as is the case with the recently launched IPv6 endpoint and the public regional endpoint

It isn't explicitly stated in the RCA but it is likely these new endpoints were the straw that broke the camel's back for the DynamoDB load balancer DNS automation

4. yla92 ◴[23 Oct 25 03:02 UTC] No.45677745[source]▶
So the root cause is basically race condition 101 stale read ?
5. galaxy01 ◴[23 Oct 25 05:18 UTC] No.45678380[source]▶
Would conditional read/write solve this? looks like some kind of stale read
6. lazystar ◴[23 Oct 25 06:01 UTC] No.45678631[source]▶
> Since this situation had no established operational recovery procedure, engineers took care in attempting to resolve the issue with DWFM without causing further issues.

interesting.

7. shayonj ◴[23 Oct 25 07:40 UTC] No.45679256[source]▶
I was kinda surprised the lack of CAS on per-endpoint plan version or rejecting stale writes via 2PC or single-writer lease per endpoint like patterns.

Definitely a painful one with good learnings and kudos to AWS for being so transparent and detailed :hugops:

8. jasode ◴[23 Oct 25 08:02 UTC] No.45679393[source]▶
So the DNS records if-stale-then-needs-update it was basically a variation of the "2 Hard Things In Computer Science - cache invalidation". Excerpt from the giant paragraph:

>[...] Right before this event started, one DNS Enactor experienced unusually high delays needing to retry its update on several of the DNS endpoints. As it was slowly working through the endpoints, several other things were also happening. First, the DNS Planner continued to run and produced many newer generations of plans. Second, one of the other DNS Enactors then began applying one of the newer plans and rapidly progressed through all of the endpoints. The timing of these events triggered the latent race condition. When the second Enactor (applying the newest plan) completed its endpoint updates, it then invoked the plan clean-up process, which identifies plans that are significantly older than the one it just applied and deletes them. At the same time that this clean-up process was invoked, the first Enactor (which had been unusually delayed) applied its much older plan to the regional DDB endpoint, overwriting the newer plan. The check that was made at the start of the plan application process, which ensures that the plan is newer than the previously applied plan, was stale by this time due to the unusually high delays in Enactor processing. [...]

It outlines some of the mechanics but some might think it still isn't a "Root Cause Analysis" because there's no satisfying explanation of _why_ there were "unusually high delays in Enactor processing". Hardware problem?!? Human error misconfiguration causing unintended delays in Enactor behavior?!? Either the previous sequence of events is considered unimportant, or Amazon is still investigating what made Enactor behave in an unpredictable way.