Especially given that end-of-support is only 18-months from now (April 2026) … when end-of-support of v5.7 is what drive them to upgrade in the first place.
Especially given that end-of-support is only 18-months from now (April 2026) … when end-of-support of v5.7 is what drive them to upgrade in the first place.
Also: 8.0 is old and most issues have been found. 8.4 probably has more unknowns.
MySQL is very unstable software - hopefully this will be past - and it's very reasonable to go for the smallest upgrade steps possible.
8 has nice features and I think they evaluated it as stable enough to upgrade their whole fleet to it. I'm pretty sure from 8 to 8.4 the upgrades will be much simpler.
"This distinction played a crucial role in our upgrade planning and execution strategy."
"Navigating Challenges in the MySQL Upgrade Journey"
"Finally, minimizing manual intervention during the upgrade process was crucial."
Really? At n=1 the rate seems to be 0
Whilst it may smell of ChatGPT/Claude, I think the answer is actually simpler.
Look at the authors of the blog, search LinkedIn. They are all based in India, mostly Bangalore.
It is therefore more likely to be Indian English.
To be absolutely clear, for absolute avoidance of doubt:
This is NOT intended a racist comment. Indians clearly speak English fluently. But the style and flow of English is different. Just like it is for US English, Australian English or any other English. I am not remotely saying one English is better than another !
If, like me, you have spent many hours on the phone to Bangalore call-centres, you will recognise many of the stylistic patterns present in the blog text.
Being able to load in a new password while the current one is active is where it's at! Trying to coordinate a big bang where everyone flips over at the same time is misery, and I spent a bunch of time updating services to not have to do that! Great enhancement.
I wonder what other datastores have dual (or more) password capabilities?
Hate to generalize, but this has less to do with "Indian style" but rather adding a lot of fluff to make a problem appear more complex than it is, OR maybe someone set a template that you must write such and such sections, despite there not being relevant content. [ Half the sections from this article could be cut without losing anything ]
In this case, the _former_ really shouldn't have been the case. I for one would love to read a whole lot more about rollback planning, traffic shifting, which query patterns saw most improvements, hardware cost optimizations, if any, etc.
pretty sure safe travels is critical to maximum value to their shareholders (aka stfu or tell me how this blog post has anything to do with maximize shareholder value https://www.uber.com/en-JO/blog/upgrading-ubers-mysql-fleet/... ... shareholder value is a dumb ass thing to prioritize over human life)
Edit; also, as can be seen here in responses, and elsewhere on the web when discussing this, the fans say it's no problem, but many less religious users feel it's a massive design flaw (perfectly logical at the time, not so logical now) that sometimes will stop users from using it, which is a shame
I'd be interested to hear if my assumptions were wrong, or if their schema and/or queries make this more intense than it seems.
That doesn't sound unreasonable, on average. But I suspect the distribution is likely pretty uneven.
We ended up spinning up a secondary fleet and bin log replicating from our 5.7 master to the to-be 8.0 master until everything made the switch over.
I was frankly surprised it worked, but it did. It went really smoothly.
how is this done - like a user would have history, payments, etc. are all of them colocated in one cluster ? (which means the sharding is based on userid) ?
is there then a database router service that routes the db query to the correct database ?
also, the US should be wayyyyy stricter on who we issue drivers license to. so many terrible drivers on the road driving these death machines.
In a like vein when Australians say "goodeye" they usually aren't talking about your vision.
It is an extension though so downside there is it not being included in most Postgres installs. I’ve used it at work and it felt like a superpower getting the benefits of a vacuum full without all the usual drama.
The biggest difference, IMO, is if you _aren't_ aware of the clustering index, and so design your schema in a way that is suboptimal. For example, given a table with orders or something similar, with a PK of `(user_id, created_at)`, a query like `SELECT... FROM... WHERE user_id = ? ORDER BY created_at DESC LIMIT 1` can be quite fast in MySQL. With a monotonic integer as the PK, and a secondary index on those two columns, it can still be quite fast, depending on insert order. If however you invert the PK to `(created_at, user_id)`, while MySQL 8+ will still be able to use the index, it's not nearly as efficient – in my tests, I saw query speed go from 0.1 msec --> 10 msec.
In contrast, while there is a small difference in Postgres with all of those differences, it's just not that big of a difference, since it stores tuples in a heap. Again, in my tests, query speed went from 0.1 msec --> 0.7 msec. This is a point query, of course; with range queries (`WHERE created_at < ...`) Postgres suffered more, jumping up to ~25 - 50 msec.
[0]: http://smalldatum.blogspot.com/2024/09/mysql-and-postgres-vs...
In a low-resource environment deferring work makes sense. But even in low-resource environment the vacuum process would consume huge amounts of resources to do its job, especially given any kind of scale. And the longer it's deferred the longer the process will take. And if you actually are in a low-resource environment it'll be a challenge to have enough disk space to complete the vacuum (I'm looking at you, sunos4) - and don't even talk about downtime.
I don't understand how large pgsql users handle vacuuming in production. Maybe they just don't do it and let the disk usage grow unbounded, because disk space is cheap compared to the aggravation of vacuuming?
I had to really think hard why it is incorrect / not common elsewhere. Had to see comments to learn -- someone explained that a native English speaker would instead say "this morning" and not "today morning".
As a Indian ESL speaker -- "today morning" sounded (and still sounds) perfectly fine to me -- since my brain grew up with indian languages where this literal phrase (equivalent of "TODAY morning") is not only very common, but also the normal/correct way to convey the idea, and if we instead try to say "THIS morning" it would feel pretty contrived.
Frankly, ChatGPT could have written this better with a simple “improve the style of this text” directive.
Example from the start:
> MySQL v8.0 offered a compelling proposition with its promise of substantial performance enhancements.
That could have just been “MySQL v8.0 promised substantial performance improvements.”
Or perhaps it was meant to specify that they, themselves, might have been presumed to be an outlier who would think it was a good idea, but who has in fact come to think that is a bad idea.
Examples of this kind of counter-presumptive use of the word "even":
1: On animals and the weather: "It was so cold that even polar bears were suffering from frostbite and frozen digits."
2: On politics, where one's general stance is well-known and who who might be rationally presumed to be a supporter of a particular thing: "Even I think that this issue is a total non-starter."
Even if they may have meant something else, that doesn't mean that they didn't intend for the words to be taken non-literally.
I don't work for Uber, but this is almost certainly the assumption that is wrong. I doubt there is just a single workload duplicated 2.1K times. Additionally, different regions likely have different load.
Short version is that VACUUM is needed to clean up dead tuples and reclaim disk space. For most cases with smaller amounts of data, auto-vacuum works totally fine. But I've had issues with tables with 100m+ rows that are frequently updated where auto-vacuum falls behind and stops working completely. These necessitated a full data dump + restore (because we didn't want to double our storage capacity to do a full vacuum). We fixed this by sharding the table and tweaking auto-vacuum to run more frequently, but this isn't stuff you have to worry about in MySQL.
Honestly if you're a small shop without database/postgres experts and MySQL performance is adequate for you, I wouldn't switch. Newer versions of MySQL have fixed the egregious issues, like silent data truncation on INSERT by default, and it's easier to maintain, in my experience.
this just makes no sense bc the drivers are on all of the different apps. rework your formula.
Vacuuming indices is a bit more finicky with locks, but you can just periodically build a new index and drop the old one when it becomes an issue
It's just different problems.
Also this paragraph sounds a lot like it has been written by LLMs, it's over-expressive:
We systematically advanced through each tier, commencing from tier 5 and descending to tier 0. At every tier, we organized the clusters into manageable batches, ensuring a systematic and controlled transition process. Before embarking on each stage of the version upgrade, we actively involved the on-call teams responsible for each cluster, fostering collaboration and ensuring comprehensive oversight.
The paragraph uses "commencing from" together with "descending to". People would probably write something like "starting with". It shows how the LLM has no spatial understanding: tier 0 is not below or above tier 5, especially as the text has not introduced any such spatial ordering previously. And it gets worse: there is no prior mention of the word "tier" in the blog post. The earlier text speaks of stages, and lists 5 steps (without giving them any name, but the standard term is more like "step" instead of "tier").There is more signs like "embark", or that specific use of "fostering collaboration" which goes beyond corporate-speak, it also sounds a lot like what an LLM would say. Apparently "safeguard" is also a word LLMs write very often.
This isn't 100% equal as ownership (thus permissions with DEFINER) in stored procedures etc. needs some thought, but bad access using outdated username is simpler to trace (as username can be logged etc. contrary to passwords; while MySQL allows for tracing using performance_schema logging incl. user defined connection attributes which may ease finding the "bad" application)
I feel like this article challenges my patience and attention too much, there is really no need to focus on the pros of upgrading here. We reader just want to know how they managed to upgrade at that large scale, challenges they faced and how the solved them. Not to mention any sane tech writers that value their time wouldn't write this much.
We don't just carry out a MySQL upgrade, oh no. We embark on a significant journey. We don't have reasons, but compelling factors. And then, we use compelling again soon after when describing how "MySQL v8.0 offered a compelling proposition with its promise of substantial performance enhancements", just as any human meatbag would.
[1] https://www.latimes.com/socal/daily-pilot/opinion/story/2024...
[2] https://english.elpais.com/science-tech/2024-04-25/excessive...
PostgreSQL tables are known as heaps, which consist of slotted pages where new data is written to the first page with sufficient free space. Since it's not a b-tree and you can't resolve a row with just a primary key without a table scan, Postgres uses the physical location of the row called a tuple ID (TID, or item pointer) that contains the page and position (slot) of the row within that page. So the TID (10, 3) tells Postgres the row is in block 10 slot 3 which can be fetched directly from the page buffer or disk without having to do a tree traversal.
When PostgreSQL updates a row, it doesn’t modify the original data directly. Instead, it:
1) Writes a new version of the row to a new page
2) Marks the old row as outdated by updating its tuple header and relevant page metadata
3) Updates the visibility map to indicate that the page contains outdated rows
4) Adjusts indexes to point to the new TID of the updated row
This means that indexes need to be updated even if the column value didn't change.Old rows continue to accumulate in the heap until the VACUUM process permanently deletes them, but this process can impact normal operations and cause issues.
Overall this means Postgres does more disk I/O for the same work as MySQL. The upside is Postgres doesn't have to worry about page splits, so things like bulk inserts can be much more efficient.
This is a big part of why the tech is so damn corrosive, even in well-meaning use, let alone its lopsided benefits for bad actors.
Even on the “small” and more-private side of life, it’s tempting to use it to e.g. spit out a polished narrative version of your bullet-point summary of your players’ last RPG session, but then do you go cut it back down to something reasonable? No, by that point it’s about as much work as just writing it yourself in the first place. So the somewhat-too-long version stands.
The result is that the temptation to generate writing that wasn’t even worth someone’s time to write—which used to act as a fairly effective filter, even if it could be overcome by money—is enormous. So less and less writing is worth the reader’s time.
As with free long distance calls, sometimes removing friction is mostly bad.
We've been using https://github.com/dataegret/pgcompacttable to clean up bloat without impacting stability/performance as much as pg_repack does.
Too bad it's sorta annoying to do on plain old pg.
I wrote a summary of the DDL / table design differences between MySQL and MariaDB, and that topic alone is fairly long: https://www.skeema.io/blog/2023/05/10/mysql-vs-mariadb-schem...
Another area with major differences is replication, especially when moving beyond basic async topologies.
“Stops working completely” should not be a thing, it could be vacuuming slower than the update frequency (although that’d be surprising) but I don’t know of any reason it’d just stop?
That being said I’ve also had issues with autovac (on aurora to be fair, couldn’t say if it was aurora-specific) like it running constantly without vacuuming anything, like there was an old transaction idling (there wasn’t)
I'm sure there are people who write like that. LLMs have to get it from somewhere. But that part especially is mostly empty phrases, and the meaning that is there isn't all that flattering
For indices, as you mentioned, doing either a REINDEX CONCURRENTLY (requires >= PG12), or a INDEX CONCURRENTLY / DROP CONCURRENTLY (and a rename if you’d like) is the way to go.
In general, there is a lot more manual maintenance needed to keep Postgres running well at scale compared to MySQL, which is why I’m forever upset that Postgres is touted as the default to people who haven’t the slightest clue nor the inclination to do DB maintenance. RDS doesn’t help you here, nor Aurora – maintenance is still on you.
Aside: I wish Postgres forced to make explicit the lock taken. Make me write “TAKE LOCK ACCESS EXCLUSIVE VACUUM FULL my_table”, and fail if the lock I take is too weak. Implicit locks are such a massive footgun that have caused countless incidents across the world, it’s just bad design.
If you think this sounds more like a job for a K/V store than a relational database, well, you'd be right; this is why e.g. Facebook moved to MyRocks. But MySQL/InnoDB does a decent job and gives you features like write guarantees, transactions, and solid replication, with low write latency and no RAFT or similar nondeterministic/geographically limited protocols.
* You can also structure your data so that the shard is encoded in the lookup key so the "routing" is handled locally. Depends on your setup
At that point, if you're reaching in and scripting your pods to do what you want, you lose a lot of the benefits of convention and reusability that k8s promotes.
Don't get me wrong, a lot of people have talked about Uber doing overengineering in weird ways, maybe they're even completely right. But being like "Well, obviously x/y = z, and z is rather small, therefore it's not impressive, isn't this obvious?" is the computer programming equivalent of the "econ 101 student says supply and demand explain everything" phenomenon. It's not an accurate characterization of the system at all and falls prey to the very thing you're alluding to ("this is obvious.")
Corporate (and SEO) writing has always been overly verbose and tried to sound fancy. In fact, this probably is where LLMs learned that style. There's no reliable heuristic to tell human- and AI-writing apart.
There's a lot of worry about people being fooled by AI fakes, but I'm also worried about false positives, people seeing "AI" everywhere. In fact, this is already happening in the art communities, with accusations flying left and right.
People are too confident in their heuristics. "You are using whole sentences? Bot!" I fear this will make people simplify their writing style to avoid the accussations, which won't really accomplish anything, because AIs already can be prompted to avoid the default word-salad style.
I miss the time before LLMs...
In any case, this got resolved but caused a huge operational headache, and isn't something that would have been a problem with MySQL. I feel like that's the main reason VACUUM gets hated on; all of the problems with it are solvable, but you only find those problems by running into them, and when you run into them on your production database it ends up somewhere between "pain in the ass" and "total nightmare" to resolve.
It continues to baffle me why AWS picks some truly terrible defaults for parameter groups. I understand most of them come from the RDBMS defaults, but AWS has the luxury of knowing precisely how many CPUs and RAM any given instance has. On any decently-sized instance, it should allocate far more memory for maintenance_work_mem, for example.
If you are up-to-date with all your libraries it all should go well, but if some project is stuck on some old code, mostly old mysql libraries, one might get surprises when doing the switch away.
At least in RDS, that will be a one-way upgrade ie: no rollback will be possible. That said, you can upgrade one instance at a time in your cluster for a no-downtime rollout.
Not in the heap, but if you have any index on the table (I know, don’t do that for bulk loads, but many don’t / it isn’t feasible sometimes) then you’re still dealing with a B+tree (probably).
Also, MySQL still gets the nod for pure bulk load speed via MySQLShell’s Parallel Import Utility [0]. You can of course replicate this in Postgres by manually splitting the input file and running multiple \COPY commands, but having a tool do it all in one is lovely.
[0]: https://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-shell-uti...
https://www.forbes.com/sites/danielnewman/2023/02/21/uber-go...
I've worked on 20+ projects using MySQL in consulting career. Not once stability was a concern. Banking clients would even routinely shut down radom MySQL nodes in production to ensure things continued running smoothly.
As I'm sure users like Uber and Youtube would agree. And these too: https://mysql.com/customers
Unless you know something we don't and we're just lucky.
The current bottleneck appears to be etcd, boltdb is just a crappy data store. I would really like to try replacing boltdb with something like sqlite or rocksdb as the data persistence layer in etcd but that is non-trivial.
You also start seeing issues where certain k8s operators do not scale either, for example cilium cannot scale past 5k nodes currently. There are fundamental design issues where the cilium daemonset memory usage scales with the number of pods/endpoints in the cluster. In large clusters the cilium daemonset can be using multiple gigabytes of ram on every node in your cluster. https://docs.cilium.io/en/stable/operations/performance/scal...
Anyways, the TL;DR is that at this scale (16k nodes) it is hard to run k8s.
Largely as @brk above you already mentioned, tendency to use formal and obscure words alongside a specific tone. I'll also re-iterate what @brk said, hard to fully describe, more of a "you know it when you see it".
If I had to pick some specific examples from the blog post, the following phrase is a good example:
We systematically advanced through each tier, commencing from tier 5 and descending to tier 0.
There are 101 ways you could write that in US English, but I reckon 99% of the US population would be unlikely to pick the above unless they were writing an academic paper or something.
This one is also quite Indian English in many respects:
Our automated alerts and monitoring system actively oversees the process to ensure a seamless transition and promptly alerts of any issues that may arise.
Similarly, we have stylistic elements such as the over-breaking of paragraphs to the extent it becomes a series of statements. For example:
Upgrading to MySQL 8.0 brought not only new features, but also some unexpected tweaks in query execution plans for certain clusters. This resulted in increased latencies and resource consumption, potentially impacting user experience. This happened for the cluster which powers all the dashboards running at Uber. To address this issue, we collaborated with Percona, identified a patch fix, and successfully implemented it for the affected clusters. The resolution ensured the restoration of optimized query performance and resource efficiency in alignment with the upgraded MySQL version.
A relatively short paragraph, but five phrases. Your average US English writer would likely word it differently resulting in it being trimmed down to two or three phrases.
As I said in my original post though, none of it is bad English, its just a different style.
Good human writing is short and to the point. (Technical writing at least.) But this is not a result of laziness — it’s actually more difficult.
“If I had more time, I would have written a shorter letter.” - Blaise Pascal, and probably others [0]
In any case I find these LLM “gotcha” comments incredibly tedious.
[0] https://quoteinvestigator.com/2012/04/28/shorter-letter/?amp...
Absolutely, though they're generally orders of magnitude smaller than the table file unless you're INCLUDE'ing lots of columns.
There's pg_bulkload which supports parallel writers as well as deferred index updates until the loading process is complete. Not sure how it compares to what MySQL offers out of the box, but I definitely agree that the MySQL tooling ecosystem in general has a leg up.
Global public cloud spend is hundreds of billions of dollars a year. I wouldn't be surprised if it's AWS's marketing team that came up with the talking point about how much more expensive developer time is.
Edit: put this another way- wherever you work, you might know what parts of the architecture need some performance work but do you know what parts of the architecture cost the most money?
I might say, "hardware" is expensive compared to (my) salary :)
Their style is much more direct if you just ask them a question or to summarize something. (Although whether the answer is accurate or not is another matter.)
LLM is the new spellchecker. Soon we'll we will wonder why some people don't use it to sanity check blog posts or any other writing.
And let's be honest, some writings would greatly benefit from a sanity check.
Care to elaborate at all? Were they more like missing edge cases or absent core functionality? Not to imply that missing edge cases aren’t important when it comes to DB ops.
There are sometimes temporary views to keep the old and new code working during gradual transitions.
I’ve tested Aurora Postgres and MySQL against both RDS and native (on my own, extremely old hardware), and Aurora has never won in performance. I’ve been told that “it’s better in high concurrency,” but IMO, that’s what connection poolers are for.
It doesn’t take engineering knowledge to browse through CloudWatch metrics and see that your average CPU utilization is in the single digits.
Another common phrase in Indian English is "do the needful," which is a delightful formulation. Grammarly has a plausible description of how it arose. [0]
Though the single threaded performance fell off a cliff, tanking my CI performance.
Maria DB evolved very differently. I'm not sure how they stack up performance wise.
For simple use cases or with tried and true patch sets I'm sure it can be a work horse.
In normal circumstances it's not needed. The unused space is reused for further inserts and if your database maintains steady rate of inserts/deletes, this won't be an issue.
However I experienced situation when database starts to grow uncontrollably and it was happening for weeks (when actual workload didn't change). I don't know what causes that behaviour. The solution was to run `vacuum full`.
Well, it is... sort of.
Obviously you can't have Uber be a guaranteed way to be robbed by a highwayman, but when you've cleared out most taxis in a given city, you can start to dictate the terms by which customers accept your service.
And if that means including language in your ToS that shove your customers into a binding arbitration agreement [0] that effectively shield you from the risks of hiring incompetent or malicious drivers, well... that's what that means.
[0]https://www.npr.org/2024/10/02/nx-s1-5136615/uber-car-crash-...
Taxi companies made it work.
They just accepted less return on their investment than the tech bros behind Uber did.
I wish it didn’t turn me off the content as much as it does but it’s very jarring.
It certainly has a drop-swap approach which does a table rename, even though the resulting table ends up with the same name as before.
That's how I encountered the MySQL table-rename-crash bug.
My definition of "very unstable" has been very vague, so I'll clarify.
MySQL is stable software if a company uses a subset of features - the core ones, stable for a long time. Anything outside of that area, and either a company has very considerable engineering resources (that is, being able to work on the MySQL source, indeed like Uber, Google or Facebook), or it's suicide.
For starters, the MySQL development model has a habit of abandoning functionalities while not officially deprecating them, leaving them buggy.
MyISAM for example has been abandoned long ago, but not officially. Around 10/12 years ago, while it was still somewhat used, we experienced a bug where the tables were marked as crashed on clean shutdowns. Very nasty because it caused long recovery times on startup.
InnoDB fulltext indexes are the same. They have a broken workflow (they require maintenance, but some required maintenance functionalities are missing), and clearly MySQL has abandoned them, but not officially deprecated.
New functionalities have typically nasty bugs. The critical bugs I've mentioned were on simple operations related to relatively new, but not widespread, functionalities (MVIs, regular expressions, InnoDB FTIs, I think JSON, and can't remember the other(s)).
Worse, MySQL 8.0 has been a complete trainwreck, because the product managers decided to develop features/changes in patch versions, continuosly introducing breakages. So, maybe a functionality that has finally been stable for an year or more, suddenly breaks (in production, of course).
Heck, even something as simple as 'SHOW TRIGGERS' is currently buggy, just because triggers are not common in the userbase.
Given the domain you've mentioned (banking) you've probably been working with a conservative set of functionalities - which absolutely makes sense, but calling MySQL stable depends on whether one considers MySQL a small set of functionalities, or the whole set of them.
I wonder because the promotion of the secondary v8 node to primary is a breaking change in this path, whereas in an incremental upgrade it might not have been. But I also understand at this sort of scale things might be as easy as that.