←back to thread

AWS Lambda Silent Crash – A Platform Failure, Not an Application Bug [pdf]

(lyons-den.com)
115 points nonfamous | 4 comments | 15 Jul 25 01:22 UTC | HN request time: 2.219s | source
1. sneilan1 ◴[15 Jul 25 03:00 UTC] No.44567482[source]▶
`This wasn’t a request for clarification. It was a reset — a request for artefacts that have already been delivered, multiple times, across prior case updates, call notes, and debug summaries. The very same package AWS previously reviewed. The same logs they had already quoted. The same failure they had already confirmed.`

While the author is probably correct in this being a platform level bug (I have not ran the code to confirm myself though), they should stop being so angry at a corporation doing what corporations do which is run a big bureaucracy and require extraordinary amounts of extra communication. One of job requirements of a principle engineer, often making hundreds of thousands if not millions of dollars a year, is to communicate across levels because it is so hard to do this effectively without being angry or losing your cool.

`As a solo engineer, I outpaced AWS’s own diagnostics, rebuilt our deployment stack from scratch, eliminated every confounding factor, and forced AWS to inadvertently reproduce a failure they still refused to acknowledge.`,

I do not think that a solo engineer would have the economic clout to get AWS to pay attention to the bug you found. Finding a bug in a system does not imply any ability to fix a bug in a bureaucracy.

Plus, given the tone of the article, I have a funny feeling that the author may have rubbed people in AWS the wrong way, preventing more progress from being made on their bug.

But, a wise fellow once said, "all progress in this world depends upon the unreasonable man".

replies(2): >>44567841 #>>44571252 #
2. ◴[15 Jul 25 04:31 UTC] No.44567841[source]▶
3. ceejayoz ◴[15 Jul 25 14:00 UTC] No.44571252[source]▶
It's not a platform level bug. It's as-designed, as-documented behavior. Return a response and Lambda suspends.

They seem to want a fast response with background processing of a potentially slow task. The answer on Lambda (and honestly, elsewhere) is having this function send the payload to a queue, and processing the queue with another function.

replies(1): >>44572321 #
4. sneilan1 ◴[15 Jul 25 15:42 UTC] No.44572321[source]▶
Yes you’re right. I wrote the parent comment and then someone else also mentioned the lambda is returning before background processing. I missed that one!