How I cut GTA Online loading times by 70%

The old maxim of "Premature optimization is the root of all evil" has over time evolved to "If you care one iota about performance, you are not a good programmer".

And fwiw, the full quote is:

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

Yet we should not pass up our opportunities in that critical 3%.

Also, it’s not “never optimise”. It’s “only optimise once you’ve identified a bottleneck”. I guess in a profit-making business you only care about bottlenecks that are costing money. Perhaps this one isn’t costing money.

ding ding ding ding

This one isn’t measured to be costing money. Trying to figure out how much money you’re losing as a result of performance problems is extremely difficult to do.

Then again, using the correct data structure is not really optimisation. I usually think of premature optimisation as unnecessary effort, but using a hash map isn't it.

That belief is getting a bit outdated now that computing efficiency is hitting walls. Even when compute is cheaper than development, you're still making a morally suspect choice to pollute the environment over doing useful work if you spend $100k/yr on servers instead of $120k/yr on coding. When time and energy saved are insignificant compared to development expense is of course when you shouldn't be fussing with performance.

I don't think the anti-code-optimization dogma will go away, but good devs already know optimality is multi-dimensional and problem specific, and performance implications are always worth considering. Picking your battles is important, never fighting them nor knowing how is not the trick.

That doesn't really apply here. I don't even play GTA V but the #1 complain I've always heard for the past 6 years is that the load times are the worst thing about the game. Once something is known to be the biggest bottleneck in the enjoyment of your game, it's no longer "premature optimization". The whole point of that saying is that you should first make things, then optimize things that bring the bring the most value. The load time is one of the highest value things you can cut down on. And the fact that these two low hanging fruit made such a big difference tells me they never gave it a single try in the past 6 years.

Perhaps this one is costing them a lot of money.

My belief is that your first goal should be cognitive optimization. I.e. make it simple and clear. That includes using hash maps when that is the proper data structure since that’s is what is called for at a base design level.

The next step is to optimize away from the cognitively optimal, but only when necessary. So yeah it’s really crazy this was ever a problem at all.

If it doesn't change semantics, it's optimization.

It's just a zero cost one, so if anybody appears complaining that you are caring to choose the correct data structure and that's premature (yeah, it once happened to me too), that person is just stupid. But not calling it an optimization will just add confusion and distrust.

This is the key point: premature optimization would be e.g. denormalising a database because you think you might have a performance problem at some point, and breaking the data model for no good reason.

Here the wrong data structure has been used in the first place.

Sure it does apply. These complaints come out after the game has been released. They should have optimized this before they released, while they even designed the system. However that's considered premature optimization, when in fact it's just bad design.

I agree 100% - the whole cheery lack of care around optimization to the point of it becoming 'wisdom' could only have happened in the artifice of the huge gains in computing power year on year.

Still, people applying optimizations that sacrifice maintainability for very little gain or increase bugs are still doing a disservice. People who understand data flow and design systems from the get-go that are optimal are where it's at.

The problem here isn't a lack of optimization, it's a lack of profiling. Premature optimization is a problem because you will waste time and create more complex code optimizing in places that don't actually need it, since it's not always intuitive what your biggest contributors to inefficiency are. Instead of optimizing right away, you should profile your code and figure out where you need to optimize. The problem is that they didn't do that.

Precisely. Hasn't GTA Online done over a billion in revenue?

Given how incredibly successful it's been, it's conceivable the suits decided the opportunity cost of investing man-hours to fix the issue was too high, and that effort would be better spent elsewhere.

They're making lots of money but the ridiculous load times absolutely cost them money. It's not worth an unlimited amount of dev time to fix, but they definitely should have assigned a dev to spend one day estimating how hard fixes would be.

Not all performance problems have obvious "bottlenecks", some are entirely second-order effects.

For instance, everyone tells you not to optimize CPU time outside of hotspots, but with memory usage even a brief peak of memory usage in cold code is really bad for system performance because it'll kick everything else out of memory.

I am not sure I would call that "evolution".

We used to start up Quake while we waited then we'd forget about GTAO. Later we'd discover GTA had kicked us out for being idle too long. Then we'd just close it.

That should be embarrassing for Rockstar but I don't think they would even notice.

That's also possible. I haven't played it myself, so I really can't comment.

I think this part of the Knuth's quote is central:

> Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered.

And he is explicitly advocating for optimizing the critical part:

> Yet we should not pass up our opportunities in that critical 3%.

And somehow, people have latched onto the catchphrase about "early optimization" that was taken out of context.

Arguably, it could actually make them money, since it provides a window of free advertising. I have no data either way, but I wouldn’t assume long load screens are necessarily bad for business.

OTOH, at some point what is good from a cognitive viewpoint depends on what idioms you use. So it can be helpful to actively chose which idioms you use and make certain to not use those that tend to blow up performance wise.

> while they even designed the system

See, that's exactly why you're wrong. This wasn't a bad "design". If the fix required to rebuild the whole thing from scratch, then you would have a point and thinking about it "prematurely" would've been a good idea. In this case, both the fixes were bugs that could've been fixed after the game was finished without having to undo much of the work done.

The whole point of the saying is that you don't know what's gonna be a bottleneck until after. Yes by optimizing prematurely, you would've caught those two bugs early, but you would've also spent time analyzing a bunch of other things that didn't need to be analyzed. Whereas if you analyze it at the end once people complain about it being slow, you're only spending the minimum amount of time necessary on fixing the things that matter.

Given the current code example, I don't even thing idioms come into it. Here a hash map data structure was called for regardless of whatever idioms come into other parts of the design. This is just fundamental and has nothing to do with functional programming, OOP, etc.

> Whereas if you analyze it at the end once people complain about it being slow

I think that we should also stop doing crash tests in cars. Just release the car to the public and analyze human crashes afterwards.

It's costing them money just in the time wasted by their own internal QAs and devs loading the game to test it.

N=1 but one of the reasons I don't partake in modern gaming is ridiculous loading times and mandatory update sizes.

I'd like to add, while GTA is running I'm really impressed by its performance, even / especially when it was on the PS3; you could drive or fly at high speed through the whole level, see for miles and never see a loading screen. It is a really optimized game, and that same level is continued in RDR2.

Which makes the persisting loading issue all the weirder.

I went back and had a look at that maxim a few years ago and found that it actually doesn't say what many people claims it says. And definitively not as the blanket excuse for slow code that it has always to some degree been used as.

The reason for the misunderstanding is that the kinds of practices it actually talks about are uncommon today. People would often take stuff written in higher level languages and reimplement them in assembler or machine code. Which makes them more time-consuming to change/evolve.

It also isn't like it is hard to figure out which part of a piece of software that is taking up your runtime these days. All worthwhile languages have profilers, these are mostly free, so there is zero excuse for not knowing what to optimize. Heck, it isn't all that uncommon for people to run profiling in production.

Also, it isn't like you can't know ahead of time which bits need to be fast. Usually you have some idea so you will know what to benchmark. Long startup times probably won't kill you, but when they are so long that it becomes an UX issue, it wouldn't have killed them to have a look.

Exactly. And not only being able to pick the correct data structure for the problem, (and possibly the correct implementation), but being able to know what is going to need attention even before a single line of code is written.

Most of my optimization work is still done with pencil, paper and pocket calculator. Well, actually, most of the time you won't even need a calculator.

You do understand that "my entertainment takes 6 minutes to load" is a very different problem from "my essential transportation kills people"? And therefore call for different approaches?

They don't need more than a full minute of advertising every load.

Putting aside the bad analogy that the other comment covers, the overall point I was making also covers doing the optimization before game release, but after the game is more or less complete. Rockstar optimizing during the final few months of playtest wouldn't be premature optimization.

Premature optimization is about trying to optimize the micro (a given function), while you don't yet have the macro (the game itself). If the function accounts for 0.1% of the performance of the final game, it doesn't matter if you make it 5x faster since that will only make the game 0.08% faster. You could've spent that time optimizing a different function that accounts for 10% of the performance, and even optimizing that function by 5% would make the game 0.5% faster, which is more impactful.

Back in the day when people talked about premature optimization it was about trivial things like people asking on stackoverflow whether a++, ++a or a += 1 is faster. It obviously is a net loss since ultimately it literally doesn't matter. If matters to you, you are already an expert in the subject and should just benchmark your code.

Where does the belief that corporations are somehow perfect at doing cost-benefit analysis come from? Has anyone worked in a place where that seemed to be the case?

We're talking about an issue that has been loudly complained about for 7 years (and I am evidence that it makes people play the game much less often than they would like to) that some person without access to the source code was able to identify and fix relatively quickly, it would be surprising if it took an FTE 1 week to find this with access to the source code.

This is one of the most profitable games ever, they could hire an entire team to track this down for half a year and it wouldn't even be noticeable on their balance sheet. And I would bet money that it would have increased their active player base (which they care about because of the micro-transactions) in a noticeable way, as long as players were aware of the improvement so they would try it again.