So you want to build your own data center

Reminds me of the old Rackspace days! Boy we had some war stories:

   - Some EMC guys came to install a storage device for us to test... and tripped over each other and knocked out an entire Rack of servers like a comedy skit. (They uh... didn't win the contract.)
   - Some poor guy driving a truck had a heart attack and the crash took our DFW datecenter offline. (There were ballards to prevent this sort of scenario, but the cement hadn't been poured in them yet.)
   - At one point we temporarily laser-beamed bandwidth across the street to another building
   - There was one day we knocked out windows and purchased box fans because servers were literally catching on fire.

Data center science has... well improved since the earlier days. We worked with Facebook on the OpenCompute Project that had some very forward looking infra concepts at the time.

Once worked in a "DC" in a converted cow shed in the English countryside. Hot takes that align with your experiences:

    - A key microwave link kept going down with intermittent packet errors way down in the data link layer. A short investigation discovered that a tree across the road had come into leaf, and a branch was blowing into the line of sight of the kit on our building. A step-ladder, a saw and 10 minutes later we restored connectivity
    - Our main (BGP-ified) router out of the DC - no, there wasn't a redundant device - kept rebooting. A quick check showed the temp in the DC was so high, cooling so poor, that the *inlet* fan had an air temp of over 60C. We pointed some fans at it as a temporary measure. 
    - In a similar vein, a few weeks later the air con in another room just gave up and started spewing water over the Nortel DMS-100 (we were a dial-in ISP with our own switch). Wasn't too happy to be asked to help mop it up (thought the water could potentially be live), but what to do?

After that experience I spent time on a small, remote island where main link to the internet was a 1MB/sec link vis GS satellite (ping times > 500ms), and where the locals dialled in over a microwave phone network rated to 9600 baud, but somehow 56k modems worked... One fix I realised I needed was a Solaris box was missing a critical .so, there were no local backups or install media and so I phoned my mate back in the UK and asked him to whack up a copy on an FTP server for me to get the box back online.

And a few years after that I also got to commission a laser beam link over Manchester's Oxford Road (at the time, the busiest bus route in Europe), to link up an office to a University campus. Fun times.

It was all terrific fun, but I'm so glad I now only really do software.

> It was all terrific fun, but I'm so glad I now only really do software.

I don't blame you, a lot of us had to do things outside the box. Could be worse though, I saw a post on r/sysadmin yesterday where a poor guy got a support ticket to spray fox urine outside near the generators.