So you want to build your own data center

I’m my experience and based on writeups like this: Google hates having customers.

Someone decided they have to have a public cloud, so they did it, but they want to keep clients away with a 3 meter pole.

My AWS account manager is someone I am 100% certain would roll in the mud with me if necessary. Would sleep in the floor with us if we asked in a crisis.

Our Google cloud representatives make me sad because I can see that they are even less loved and supported by Google than us. It’s sad seeing someone trying to convince their company to sell and actually do a good job providing service. It’s like they are setup to fail.

Microsoft guys are just bulletproof and excel in selling, providing a good service and squeezing all your money out of your pockets and you are mortally convinced it’s for your own good. Also have a very strange cloud… thing.

As for the railway company going metal, well, I have some 15 years of experience with it. I’ll never, NEVER, EVER return to it. It’s just not worth it. But I guess you’ll have to discover it by yourselves. This is the way.

You soon discover what in freaking world is Google having so much trouble with. Just make sure you really really love and really really want to sell service to people, instead of building borgs and artificial brains and you’ll do 100x better.

My AWS account manager took me fishing. That’s what you get for a >$1M/yr spend. I don’t sense they would roll in mud with me, which is kind of incredible. I wonder how much you need to spend to get into mud rolling territory?

AWS support in general is extremely good in my experience. (We pay for whatever the tier below Enterprise is called, I think it costs 10% of your spend)

I’ve been on 4 hour screenshare with AWS engineers working through some infrastructure issues in the past, and we only spend $100k/yr.

Even at the $100k/yr spend level, AWS regularly reaches out with offers to try new services they’re launching for free. We’ve said “sure” a couple times, and AWS shows up with 4-6 people on their end of the call (half of them engineers).

In the past 10 years, we’ve had maybe 2-3 emergency issues per year, and every time I’m able to get a really smart person on a call within 5 minutes.

This is the #1 thing I’d be concerned about losing if we did colo or bare metal with cheaper providers.

My experience with AWS support has been downright freaky.

With other vendors, when I call a support line with an obscure issue that maybe only I hit in the whole world I fully expect to explain it to an overseas call centre drone with a poor voice line and rudimentary English. Then I expect to have to repeatedly escalate over months and be told “We can’t reproduce this glaringly obvious bug, closed.” That’s ignoring the series of very closely related family of issues I dug up in the process of troubleshooting. Which they continue to ignore because it’s “out of scope” for the ticket. “Open a new ticket and go through the pain again, peasant!”

With AWS my experience has always been “I’ve fixed that right up for you, is there anything else you’d like help with?”. Often after mere minutes!

I’m usually left speechless, ready to go on a well-practiced tirade of how “We’re spending millions on this crap and none of it works properly!”, but instead I just sit there gawping like a fish out of water, stammer “No, thank you, that was it.” and hang up in shame.

I just don’t understand why no other enterprise on Earth seems to have support this good. What’s the thing holding them back!? Maybe they assume that good support works only for this tiny upstart org called Amazon that will clearly never amount to anything!

What kind of issues you had that they could fix them immediately? I assume this is not about configuration issues on your part, but maybe I’m mistaken

I was one of the first users of the AWS Elastic File System because I had an ideal use-case for it exactly when it was first introduced. Everything worked just fine for 30 days, and then the web site basically locked up. It turned out that EFS had an initial "grace period" during which IOPS were unlimited, then it would become proportional to the GB of stored data. We had just a few hundred megabytes, so it worked out to something like 0.4 IOPS. Slower than a floppy drive! Support immediately reset the grace period for us, flipped some internal flag to make it not expire, and then a few months later the product itself was fixed to have a reasonable minimum IOPS per account irrespective of the data volume. At the time there were zero mentions of any of this on Google, I must have been the first few people to hit this issue after general availability.

A direct comparison is a nearly identical issue with Azure SQL Server Managed Instance. It too had IOPS proportional to the size of each file of a database. We migrated a database that used many small partitions (per month I think?), each with its own small file. Its performance was horrendous, easily 100x slower than on-prem. The support team could barely speak English, got repeatedly confused about the product (Azure SQL Database != SQL Managed Instance), couldn't understand the problem, and even insisted that totally broken performance "was a feature" and we should redesign "our database". Sure buddy, I'll go tell the third-party vendor that, meanwhile Microsoft themselves insisted we should migrate all of our legacy databases to this garbage. We did, it took months, cost a ton of money, and now it basically doesn't work! We abandoned the product, as have many other people. At the time, this had been an issue for many years with Microsoft engineering basically whistling at the ceiling as they cheerfully ignored it. More than a year later they fixed it, but you've got to wonder what else is still wrong with it that support can't help with.

There's more examples, but that pair stuck in my mind because they had the same root cause but wildly different outcomes.