Upgrading Uber's MySQL Fleet

Why upgrade to v8.0 (old LTS) and not v8.4 (current LTS)?

Especially given that end-of-support is only 18-months from now (April 2026) … when end-of-support of v5.7 is what drive them to upgrade in the first place.

https://en.m.wikipedia.org/wiki/MySQL

> Several compelling factors drove our decision to transition from MySQL v5.7 to v8.0:

Edit: for the downvoters, the parent comment was initially a question.

Since direct upgrade to 8.4 isn't supported. They got to go to 8.0 first.

Also: 8.0 is old and most issues have been found. 8.4 probably has more unknowns.

According to the article they started the project in 2023. Given that 8.4 was released in April 2024, that wasn't even an option when they started.

The upgrade initiative started somewhere in 2023 according to the article.

MySQL 8.4 was released in April 30, 2024.

Their criteria for a "battle tested" MySQL version is probably much more rigorous than the average CRUD shop.

I suppose they opted for a conservative upgrade policy, as v8.4 probably includes all the functional additions/changes of the previous v8.1+ versions, and moving to it would have been a very big step.

MySQL is very unstable software - hopefully this will be past - and it's very reasonable to go for the smallest upgrade steps possible.

With all the migration code already written and experience gained, I imagine upgrading 8->8.4 would take 1/10 of effort of 5.7->8.0.

Considering several versions of 8.0 had a crashing bug if you renamed a table, waiting is probably the right choice.

MySQL 8 and beyond has been riddled with bugs and performance regressions. It was a huge rewrite from 5.7.

8 has nice features and I think they evaluated it as stable enough to upgrade their whole fleet to it. I'm pretty sure from 8 to 8.4 the upgrades will be much simpler.

They started in 2023. v8.0 was the current LTS when they started.

You’re not renaming tables when you’re at scale.

Sure you do! It's how online schema changes tend to be done, e.g. https://docs.percona.com/percona-toolkit/pt-online-schema-ch... describes doing an atomic rename as the last step.

> MySQL is very unstable software

I've worked on 20+ projects using MySQL in consulting career. Not once stability was a concern. Banking clients would even routinely shut down radom MySQL nodes in production to ensure things continued running smoothly.

As I'm sure users like Uber and Youtube would agree. And these too: https://mysql.com/customers

Unless you know something we don't and we're just lucky.

You aren't renaming tables at scale because there are 27 downstream services that will break if you even think about fixing the name of the revnue_dolars table, and it's not in anyone's OKR to fix it

I’m not sure if this is a joke (ref to OKR was funny, for example), or just naive and not understanding the parent comment. I found it funny either way though.

Take a closer look at how some min downtime tools work. They often use a drop-swap strategy to replace an old table name with a new one that has schema changes.

There are sometimes temporary views to keep the old and new code working during gradual transitions.

Depends on your use case. I had a huge table get corrupted after a drop-swap rename during some DDL. 8.0.14 was the version, IIRC.

For simple use cases or with tried and true patch sets I'm sure it can be a work horse.

You clearly have not worked at global companies with concerns that go beyond “can I do it?”. I’ve used percona and I’ve used these tools before and percona is great. But when you’re at scale something as trivial as a table name change is not something you ever do because it’s not worth the risk of taking your entire site down. Just use the existing name no matter how bad it is.

Are we talking about the same Pt-online-ddl-change?

It certainly has a drop-swap approach which does a table rename, even though the resulting table ends up with the same name as before.

That's how I encountered the MySQL table-rename-crash bug.

I've used MySQL for almost 15 years, and reported dozens of bugs, a few of which were critical security issues.

My definition of "very unstable" has been very vague, so I'll clarify.

MySQL is stable software if a company uses a subset of features - the core ones, stable for a long time. Anything outside of that area, and either a company has very considerable engineering resources (that is, being able to work on the MySQL source, indeed like Uber, Google or Facebook), or it's suicide.

For starters, the MySQL development model has a habit of abandoning functionalities while not officially deprecating them, leaving them buggy.

MyISAM for example has been abandoned long ago, but not officially. Around 10/12 years ago, while it was still somewhat used, we experienced a bug where the tables were marked as crashed on clean shutdowns. Very nasty because it caused long recovery times on startup.

InnoDB fulltext indexes are the same. They have a broken workflow (they require maintenance, but some required maintenance functionalities are missing), and clearly MySQL has abandoned them, but not officially deprecated.

New functionalities have typically nasty bugs. The critical bugs I've mentioned were on simple operations related to relatively new, but not widespread, functionalities (MVIs, regular expressions, InnoDB FTIs, I think JSON, and can't remember the other(s)).

Worse, MySQL 8.0 has been a complete trainwreck, because the product managers decided to develop features/changes in patch versions, continuosly introducing breakages. So, maybe a functionality that has finally been stable for an year or more, suddenly breaks (in production, of course).

Heck, even something as simple as 'SHOW TRIGGERS' is currently buggy, just because triggers are not common in the userbase.

Given the domain you've mentioned (banking) you've probably been working with a conservative set of functionalities - which absolutely makes sense, but calling MySQL stable depends on whether one considers MySQL a small set of functionalities, or the whole set of them.