Why Companies Are Ditching the Cloud: The Rise of Cloud Repatriation

Kjell's Law: the cost of a platform eventually exceeds the cost of the one it replaced. But each cost is in a different budget.

We seem to have replaced cooling and power and a grumpy sysadmin with storage and architects and unhappy developers.

We had happy developers before? Amazing.

without the grumpy sysadmin they jump out more.

I've never worked in a data center that did cooling and power correctly. Everyone thinks they're doing it right, and then street power gets cut - there's significant impact, ops teams scramble to contain, and finally there's the finger-pointing.

I mean; it's impossible to plan for everything, and I'd argue that if you actually did plan for everything; it would be so extraordinarily overbuilt that it couldn't be considered 'correct'.

> then street power gets cut

Or the electrician doing maintenance on the backup generator doesn't properly connect the bypass and no one notices until he disconnects the generator and the entire DC instantly goes quiet.

Or your DC provisions rack space without knowing which servers are redundant with which other servers, and suddenly when two services go from 10% CPU use to 100% CPU across ten servers the breaker for that circuit gives up entirely and takes down your entire business.

The colo I’m used to has survived multiple switch overs to backup and then to diesel generators without a blip that I could detect.

I say “I’m used to” because having things there has spanned more than one job.

One power outage was days to a week. Don’t recall exactly.

It’s possible to do it right.

They are Grumpy because now they are doing Sysadmin stuff

The first time we tested cutting the power back in the day, the backup generator didn't fire! Turns out someone had pushed the big red stop button, which remains pushed in until reset.

That would have been a major problem if we'd had a nighttime power outage.

After that we ran regular switchover testing :)

The other time we ran into trouble was after someone drove a car into the local power substation. Our systems all ran fine for the immediate outage, but the power company's short term fix was to re-route power, which caused our voltage to be low enough for our UPS batteries to slowly drain without tripping over to the generator.

That was a week or two of manually pumping diesel into the generator tank so we could keep the UPS batteries topped up.

Yes it's possible. But it's not cheap. If you buy a bunch of UPS and a few generators (you need more than one, in case it doesn't start) and don't maintain them regularly and test them regularly that's when you get some bad surprises.

I have, and least power. I worked in a DC with 4 very large diesel generators, each backed up by a multi-ton flywheel that managed the transition between a power cut and the generators taking over.

Area wide power cut, winter afternoon so it was already getting dark. The two signs I knew there was something wrong were that all the lights went out outside, ie other businesses, street lighting etc. And my internet connection stopped working. Nothing else in the DC was affected. Even the elevator was working.

Amazing story, and now the flywheel is living rent free in a steampunk-inspired corner of my brain. What are these called so I can look them up on the net? Like this maybe?

https://www.torus.co/torus-flywheel

They're chonky devices which were not really off-the-shelf until the last decade, as far as I know. There's very few images of them, but a smaller one looks like this: https://www.pv-magazine.com/2018/03/14/pilot-project-for-fly...

Very cool. Somehow I missed the idea of flywheels used to store energy. I assumed one that small would peter out in seconds.

These were huge. In fact iirc the power that came into the building spun them, and the DC ran off the resultant generated electricity. So the risk at cutover is minimal, there is in fact no loss of electricity unless the wheel drops below the required revs.

One of these units blew at one point. We had 4 and only needed two running, so no big deal. The company who managed the whole thing (Swiss) came to replace it. Amazing job, they had to put it on small rollers, like industrial roller skates, then embed hooks in the walls at each corridor junction, and slowly winch the thing along, it was like watching the minute hand of a clock.

Then the whole process in reverse to bring in the new one. Was fascinating to watch. The guy in charge was a giant, built like a brick outhouse. They knew their stuff.

That whole story is clutch. Thanks a ton.

Much smaller-scale, but I worked at a company with a mini-mainframe-type computer (VAX-11/780, iirc) that had a 'motor-generator' to run it (really a motor-flywheel-generator).

The computer, storage, etc. ran off the generator, which first eliminated any risk of power spikes and surges (as the flywheel is a very effective low-pass filter), and the circuits controlling motor speed also ensured the AC frequency was better than the power company supply. This was located in a rural area, so the long power lines with few sinks (customers pulling power) made lightening spike risk spread further, and the rural voltage and frequency fluctuated a lot. Seemed like a really cool system that worked flawlessly in the years I was there.

Elegant! Flywheel as line conditioner. Most cool.

I'm realising, with my limited understanding of electronics, that the flywheel acts in these cases as a capacitor, albeit a frickin' huge mechanical one.