←back to thread

How I cut GTA Online loading times by 70%

(nee.lv)
3883 points kuroguro | 1 comments | 28 Feb 21 19:38 UTC | HN request time: 0s | source
Show context
breakingcups ◴[28 Feb 21 20:25 UTC] No.26296724[source]▶
It is absolutely unbelievable (and unforgivable) that a cash cow such as GTA V has a problem like this present for over 6 years and it turns out to be something so absolutely simple.

I do not agree with the sibling comment saying that this problem only looks simple and that we are missing context.

This online gamemode alone made $1 billion in 2017 alone.

Tweaking two functions to go from a load time of 6 minutes to less than two minutes is something any developer worth their salt should be able to do in a codebase like this equipped with a good profiler.

Instead, someone with no source code managed to do this to an obfuscated executable loaded with anti-cheat measures.

The fact that this problem is caused by Rockstar's excessive microtransaction policy (the 10MB of JSON causing this bottleneck are all available microtransaction items) is the cherry on top.

(And yes, I might also still be salty because their parent company unjustly DMCA'd re3 (https://github.com/GTAmodding/re3), the reverse engineered version of GTA III and Vice City. A twenty-year-old game. Which wasn't even playable without purchasing the original game.)

replies(40): >>26296812 #>>26296886 #>>26296970 #>>26297010 #>>26297087 #>>26297123 #>>26297141 #>>26297144 #>>26297184 #>>26297206 #>>26297323 #>>26297332 #>>26297379 #>>26297401 #>>26297448 #>>26297480 #>>26297806 #>>26297961 #>>26298056 #>>26298135 #>>26298179 #>>26298213 #>>26298234 #>>26298624 #>>26298682 #>>26298777 #>>26298860 #>>26298970 #>>26299369 #>>26299512 #>>26299520 #>>26300002 #>>26300046 #>>26301169 #>>26301475 #>>26301649 #>>26301961 #>>26304727 #>>26305016 #>>26311396 #
xyst ◴[28 Feb 21 20:35 UTC] No.26296812[source]▶
is it really unbelievable? companies this big tend to prioritize hiring a shit ton of middlemen (VPs, project managers, developer managers, offshore managers) in order to avoid paying out for talent to build and constantly maintain the project. I guess paying a shit ton of money to 1 person to manage 10+ poorly paid contractors works out for them, accounting wise.

If one really examined the accounting for GTAO, I would bet that most of the billions of dollars that were earned in micro transactions went to marketing, product research, and to middle management in the form of bonuses.

replies(3): >>26296937 #>>26296939 #>>26296958 #
IshKebab ◴[28 Feb 21 20:49 UTC] No.26296939[source]▶
It's kind of hard to believe. GTA5's online mode is their cash cow, and 6 minute load times are common?! It's kind of amazing people even play it with those load times. It's such a simple problem that one dev could have found and fixed it within a day.
replies(1): >>26297225 #
toast0 ◴[28 Feb 21 21:27 UTC] No.26297225[source]▶
It's not at all hard to believe if you've been playing video games for a while.

Everything is getting slower and slower, and nobody cares.

When I played the Atari 2600, I had to wait for the TV to warm up, but otherwise there were no games with anything approaching load times (with 128 bytes of RAM in the console, who would know). The NES didn't have much in the way of load times either, but you did have to fiddle with the cartridge slot. SNES and Genesis didn't usually load (Air Buster being a noticeable exception). CD based systems sure did like to load, but that's somewhat understandable. In the mean time, more and longer boot screens. The Switch uses a cartridge (or system flash/SD cards), but it likes to load forever too.

PC Gaming has had loading for longer, but it's been getting longer and longer.

Some arcade games have lengthy loading sequences, but only when you turn them on while they read their entire storage into RAM so they can be fast for the rest of the time they're on (which in arcades is usually all day).

replies(5): >>26297382 #>>26297857 #>>26298176 #>>26298184 #>>26313627 #
iknowstuff ◴[28 Feb 21 23:36 UTC] No.26298176{3}[source]▶
Direct Storage will allow for hardware-accelerated decompression straight from an NVMe into GPU memory, without involving the CPU and system RAM.

https://devblogs.microsoft.com/directx/directstorage-is-comi...

replies(2): >>26298411 #>>26300587 #
wtallis ◴[01 Mar 21 00:19 UTC] No.26298411{4}[source]▶
The more important thing about DirectStorage is probably that it will encourage games to use multithreaded async IO rather than serializing all their IO requests even when the underlying storage device requires dozens of simultaneous requests to deliver its full throughput.
replies(2): >>26298920 #>>26299414 #
astrange ◴[01 Mar 21 03:29 UTC] No.26299414{5}[source]▶
Does it need "multithreaded async IO" or just "async IO"? It's usually async _or_ multithreaded; the native multi-request I/O APIs are single threaded, but if you have multithreaded I/O using simpler APIs, the system is batching them into one request at the cost of a little latency.
replies(2): >>26300964 #>>26305306 #
1. wtallis ◴[01 Mar 21 09:19 UTC] No.26300964{6}[source]▶
NVMe is natively a multi-queue storage protocol, so there's no reason for the application or OS to collect IO requests into a single thread before issuing them to the lower layers. The normal configuration is for each CPU core to be allocated its own IO queue for the drive. But multithreaded synchronous (blocking) IO often isn't enough to keep a high-end NVMe SSD properly busy; you run out of cores and get bogged down in context switching overhead at a few hundred thousand IOPS even with a storage benchmark program that doesn't need any CPU time left over for productive work.

With a sufficiently low overhead async API (ie. io_uring) you can saturate a very fast SSD with a single thread, but I'm not sure it would actually make sense for a game engine to do this when it could just as easily have multiple threads independently performing IO with most of it requiring no synchronization between cores/threads.