←back to thread

Microsoft said it lost weeks of security logs for its customers' cloud products

(techcrunch.com)
285 points alephnerd | 3 comments | 20 Oct 24 21:42 UTC | HN request time: 0s | source
Show context
locusofself ◴[21 Oct 24 04:03 UTC] No.41900577[source]▶
There really was a bug in an application that just about every team runs on their VMs (simplifying here) that pushes application logs to storage. Even my team had to restart processes to get logs going again. It was a "sev 0" incident - an oopsie that was not easy to fix without many, many teams taking manual steps to restart agents which normally just hum along in the background.
replies(2): >>41903460 #>>41904064 #
1. justinclift ◴[21 Oct 24 13:39 UTC] No.41904064[source]▶
> an oopsie that was not easy to fix

Wouldn't it be nice if MS actually did automated testing to a reasonable depth, so stuff like this wouldn't keep happening?

The recent ClownStrike global outage showed a lack of testing before deployment (by ClownStrike).

This latest MS problem just demonstrates it's happening at the source (of Windows) too. It's not a good look.

replies(1): >>41905226 #
2. thewebguyd ◴[21 Oct 24 15:28 UTC] No.41905226[source]▶
This is an industry-wide problem, not exclusive to Microsoft. I feel like everyone has just outsourced QA to users. There's been a drastic decline in software quality at release, particularly in the past year-year and a half.

Initially I thought maybe it was just getting difficult to maintain these behemoth platforms that have been around since the 90s but it's infected the gaming industry as well, total green field projects where you can expect the v1.0 release to be almost unusable until 1.1 or 1.2+

replies(1): >>41905938 #
3. eitally ◴[21 Oct 24 16:46 UTC] No.41905938[source]▶
I think a lot of it has to do with how little software -- even enterprise software -- is actually written from the ground up. Reliance on both external libraries and modules owned by unrelated internal teams has made a lot of both the programming and debugging almost black box, where effective testing isn't really tractable.