Most active commenters
  • crazygringo(3)

←back to thread

How I cut GTA Online loading times by 70%

(nee.lv)
3883 points kuroguro | 17 comments | 28 Feb 21 19:38 UTC | HN request time: 1.071s | source | bottom
Show context
breakingcups ◴[28 Feb 21 20:25 UTC] No.26296724[source]▶
It is absolutely unbelievable (and unforgivable) that a cash cow such as GTA V has a problem like this present for over 6 years and it turns out to be something so absolutely simple.

I do not agree with the sibling comment saying that this problem only looks simple and that we are missing context.

This online gamemode alone made $1 billion in 2017 alone.

Tweaking two functions to go from a load time of 6 minutes to less than two minutes is something any developer worth their salt should be able to do in a codebase like this equipped with a good profiler.

Instead, someone with no source code managed to do this to an obfuscated executable loaded with anti-cheat measures.

The fact that this problem is caused by Rockstar's excessive microtransaction policy (the 10MB of JSON causing this bottleneck are all available microtransaction items) is the cherry on top.

(And yes, I might also still be salty because their parent company unjustly DMCA'd re3 (https://github.com/GTAmodding/re3), the reverse engineered version of GTA III and Vice City. A twenty-year-old game. Which wasn't even playable without purchasing the original game.)

replies(40): >>26296812 #>>26296886 #>>26296970 #>>26297010 #>>26297087 #>>26297123 #>>26297141 #>>26297144 #>>26297184 #>>26297206 #>>26297323 #>>26297332 #>>26297379 #>>26297401 #>>26297448 #>>26297480 #>>26297806 #>>26297961 #>>26298056 #>>26298135 #>>26298179 #>>26298213 #>>26298234 #>>26298624 #>>26298682 #>>26298777 #>>26298860 #>>26298970 #>>26299369 #>>26299512 #>>26299520 #>>26300002 #>>26300046 #>>26301169 #>>26301475 #>>26301649 #>>26301961 #>>26304727 #>>26305016 #>>26311396 #
crazygringo ◴[01 Mar 21 00:53 UTC] No.26298624[source]▶
I imagine the conversation between the programmer(s) and management went exactly like this:

Management: So, what can we do about the loading times?

Programmer(s): That's just how long it takes to load JSON. After all, the algorithm/function couldn't be more straightforward. Most of the complaints are probably coming from older hardware. And with new PC's and next-gen consoles it probably won't be noticeable at all.

Management: OK, guess that's that then. Sucks but nothing we can do.

Management had no idea of knowing whether this is true or not -- they have to trust what their devs tell them. And every time over the years someone asked "hey why is loading so slow?" they get told "yeah they looked into it when it was built, turns out there was no way to speed it up, so not worth looking into again."

And I'm guessing that while Rockstar's best devs are put on the really complex in-game performance stuff... their least experienced ones are put on stuff like... loading a game's JSON config from servers.

I've seen it personally in the past where the supposedly "easy" dev tasks are given to a separate team entirely, accountable to management directly, instead of accountable to the highly capable tech lead in charge of all the rest. I've got to assume that was basically the root cause here.

But I agree, this is incredibly embarrassing and unforgiveable. Whatever chain of accountability allowed this to happen... goddamn there's got to be one hell of an internal postmortem on this one.

replies(2): >>26298742 #>>26300530 #
1. CountHackulus ◴[01 Mar 21 01:11 UTC] No.26298742[source]▶
I can pretty much guarantee that there was no discussion with management like that. From experience, live ops games are essentially a perpetually broken code base that was rushed into production, then a breakneck release schedule for new features and monetization. I've personally had this conversation a few times:

Programmer: Loading times are really slow, I want to look into it next sprint.

Management: Feature X is higher priority, put it in the backlog and we'll get to it.

replies(7): >>26298776 #>>26298779 #>>26299089 #>>26299398 #>>26299585 #>>26305946 #>>26307872 #
2. sillysaurusx ◴[01 Mar 21 01:18 UTC] No.26298776[source]▶
Former gamedev here. I can vouch for this conversation. The idea that management would ask about slow loading times doesn't seem to be too realistic in my experience.
3. crazygringo ◴[01 Mar 21 01:19 UTC] No.26298779[source]▶
Normally I'd agree with you but this particularly problem is SO visible, and experienced by everyone, that I have to think management must have looked into it. I mean, it's the loading screen. They couldn't not encounter it themselves, every time.

But I could be wrong. I've only worked with sites and apps not gaming.

replies(2): >>26298811 #>>26299441 #
4. sagarm ◴[01 Mar 21 01:25 UTC] No.26298811[source]▶
Do you really think management at Rockstar is using their product?
replies(2): >>26300599 #>>26316208 #
5. dec0dedab0de ◴[01 Mar 21 02:28 UTC] No.26299089[source]▶
Ughh Every time I want to fix anything I have to sneak it in with something else I'm working on, or wait so long to get approval that I forgot all the details.
replies(1): >>26307940 #
6. Shish2k ◴[01 Mar 21 03:27 UTC] No.26299398[source]▶
At my last job I had that conversation several times :( Our website would regularly take 30s+ for a page to load, and we had an hour of scheduled downtime each week, because that’s how long it took the webapp to restart each time code was pushed. “Scheduled downtime doesn’t count as downtime, we still have the three 9’s that meets our SLA, and there’s nothing in the SLA about page load times. Now get back to building that feature which Sales promised a client was already finished”...

Aside from being generally shameful, the real kicker was that this was a "website performance & reliability" company x__x

replies(1): >>26299882 #
7. danbolt ◴[01 Mar 21 03:34 UTC] No.26299441[source]▶
I think someone in QA likely might have the scenario you describe, but I think management might only be playing the game 2-3 times a day at most. They're likely trying to keep the studio organized enough to have the game ship decently.
8. trissylegs ◴[01 Mar 21 04:02 UTC] No.26299585[source]▶
I once did get to look an optimsing a bad load time... only because at that point it was crashing due to running out of memory. (32-bit process)

In this case it was a JSON, but using .NET so Newtonsoft is at lease efficient. The issues were many cases of: * Converting string to lowercase to compare them as the keys were case insensitive. (I replaced with a StringComparison.OrderinalCaseInsensitve) * Reduntant Dictionary's * Using Dictionary<Key, Key> as a Set. replaced with HashSet.

The data wasn't meant to be that big, but when a big client was migrated it ended up with 200MB of JSON. (If there data was organised differently it would've been split accross many json blobs instead)

It would also be nice to handle it alls as UTF8 like System.Text.Json does. That would half all the strings saving a fair bit. (I mean the JSON blob it starts with is converted to UTF-16 because .NET)

9. StillBored ◴[01 Mar 21 04:58 UTC] No.26299882[source]▶
Reminds me of working for a company in the 1990's that ran constant television ads convincing everyone their experts could fix everyone's networking problems. OTOH, as an engineer working at the company the file share used for editing code/building/etc, died for an hour or two seemingly every day when the network took its daily vacation.

Many of us, after having run out of other crap to do, would sit around and wonder if the "B" grade network engineers were assigned to run the company LAN, or the ones we sent onsite were as incompetent.

replies(1): >>26299929 #
10. dragonwriter ◴[01 Mar 21 05:08 UTC] No.26299929{3}[source]▶
> Many of us, after having run out of other crap to do, would sit around and wonder if the "B" grade network engineers were assigned to run the company LAN

Internal IT is almost invariably a cost center, the technicians providing the service you are selling to customers are working in a profit center. So, yeah, probably that plus be managed in a way which focussed on minimizing cost not maximizing internal customer satisfaction or other quality metrics.

replies(1): >>26313559 #
11. sakarisson ◴[01 Mar 21 08:08 UTC] No.26300599{3}[source]▶
Their children are, at least.
12. toomanybeersies ◴[01 Mar 21 18:12 UTC] No.26305946[source]▶
I had that exact conversation at my old job. They only started listening to me when some requests started timing out because of Heroku's 30 second request duration limit.

Same thing happened with memory consumption. The problem was ignored until we regularly had background jobs using > 4 GB of memory, causing cascading failures of our EC2 instances and a whole bunch of strange behaviour as the OOM killer sniped random processes.

13. nomel ◴[01 Mar 21 20:21 UTC] No.26307872[source]▶
I experienced this to incredible extremes. It was taking our clients two hours to load their data into an external tool that they use, daily, to view that data. This was caused by our use of a data store that was only half supported by that tool. I showed that if we took two weeks to switch to the other data store, their load times would be 30 seconds. It took two years, and a new manager, to get that implemented. Unfortunately, most of the clients are still using the old data store. They haven't had time to switch to the new version...
14. nomel ◴[01 Mar 21 20:27 UTC] No.26307940[source]▶
I stopped doing this after realizing that all the extra work was just taking time away from my family. The last time I did anything out of work hours was when I told my manager that I was doing some optimization in my free time and he responded "I don't want you working on that in your free time, you could be working on this instead!". I don't plan on doing anything out of work hours again. He left, and now we're hiring more people, instead, since I can't get everything done. Win for everyone.
replies(1): >>26310641 #
15. dec0dedab0de ◴[02 Mar 21 01:02 UTC] No.26310641{3}[source]▶
Oh i never work outside of work hours. I meant that I'll just include fixes to things that are tangentially related to my current user story, or even sometimes not related at all. It helps that the team implicitly trusts my code, so anytime I say "oh and I also fixed this", it's fine. The problem is that if I say "oh I want to fix this" it gets put in the system, and ranked, and just kills my momentum.
16. imtringued ◴[02 Mar 21 09:48 UTC] No.26313559{4}[source]▶
Just because you are minimizing costs doesn't mean you have to minimize costs until you no longer get the quality you need.
17. crazygringo ◴[02 Mar 21 15:24 UTC] No.26316208{3}[source]▶
Do you really think they're not?

I don't think you choose to become a manager at one of the world's top video game companies if you don't love video games.

Hell, I don't think one of the world's top video game companies would hire you as a manager if you didn't convince them, in interviews, that you understand and love their products by using them yourself.

Using your own company's products whenever possible is generally a pretty important part of being a manager.