Most active commenters
  • jack_riminton(4)
  • buran77(4)

←back to thread

3883 points kuroguro | 43 comments | | HN request time: 2.021s | source | bottom
Show context
breakingcups ◴[] No.26296724[source]
It is absolutely unbelievable (and unforgivable) that a cash cow such as GTA V has a problem like this present for over 6 years and it turns out to be something so absolutely simple.

I do not agree with the sibling comment saying that this problem only looks simple and that we are missing context.

This online gamemode alone made $1 billion in 2017 alone.

Tweaking two functions to go from a load time of 6 minutes to less than two minutes is something any developer worth their salt should be able to do in a codebase like this equipped with a good profiler.

Instead, someone with no source code managed to do this to an obfuscated executable loaded with anti-cheat measures.

The fact that this problem is caused by Rockstar's excessive microtransaction policy (the 10MB of JSON causing this bottleneck are all available microtransaction items) is the cherry on top.

(And yes, I might also still be salty because their parent company unjustly DMCA'd re3 (https://github.com/GTAmodding/re3), the reverse engineered version of GTA III and Vice City. A twenty-year-old game. Which wasn't even playable without purchasing the original game.)

replies(40): >>26296812 #>>26296886 #>>26296970 #>>26297010 #>>26297087 #>>26297123 #>>26297141 #>>26297144 #>>26297184 #>>26297206 #>>26297323 #>>26297332 #>>26297379 #>>26297401 #>>26297448 #>>26297480 #>>26297806 #>>26297961 #>>26298056 #>>26298135 #>>26298179 #>>26298213 #>>26298234 #>>26298624 #>>26298682 #>>26298777 #>>26298860 #>>26298970 #>>26299369 #>>26299512 #>>26299520 #>>26300002 #>>26300046 #>>26301169 #>>26301475 #>>26301649 #>>26301961 #>>26304727 #>>26305016 #>>26311396 #
1. thrwyoilarticle ◴[] No.26297961[source]
The popular view is that companies who write software know how to prioritise, so if a problem like this isn't fixed, it's because they've done the calculations and decided it's not worthwhile.

I disagree. If there are no internal incentives for the people who know how to fix this to fix it, or if there's no path from them thinking fixing it could improve revenues to being assigned the ticket, things like this won't get fixed. I can fully believe the load times will result in fewer users and lower expenditure.

I think we'll see this happen with Facebook Messenger. Both the apps and the website have become slow and painful to use and get worse every month. I think we'll start to see engagement numbers dropping because of this.

replies(9): >>26298540 #>>26298890 #>>26299005 #>>26299267 #>>26299384 #>>26299942 #>>26302435 #>>26303488 #>>26304643 #
2. saghm ◴[] No.26298540[source]
> I think we'll see this happen with Facebook Messenger. Both the apps and the website have become slow and painful to use and get worse every month.

The messenger website has been atrocious for me lately. On my high-powered desktop, it often lags a bit, and on my fairly high-end laptop, it's virtually unusable. I thought it must be something I changed in my setup, but it's oddly comforting to hear that I'm not the only one with such issues.

replies(2): >>26298595 #>>26299011 #
3. baby ◴[] No.26298595[source]
For me it’s google maps, it has gotten so freaking slow both on mobile and on desktop. Actually google docs and sheet are the same.
replies(1): >>26298774 #
4. amluto ◴[] No.26298774{3}[source]
Gmail. Typing in gmail is painfully laggy.
5. rsj_hn ◴[] No.26298890[source]
That the process breaks down in some cases doesn't mean they don't know how to prioritize. They clearly know how to prioritize well enough to make a wildly successful and enjoyable game. That doesn't mean no bad decisions were made over the last decade or so of development.

Like anything else, things will be as bad as the market allows. So I'd expect monopolies to do a worse and worse job of making good decisions and companies in competitive fields to do a better and better job over time. Thus the difference between TakeTwo and Facebook, and the need for lower cost of entry and greater competition in all economic endaevors where efficiency or good decision making is important.

replies(1): >>26299314 #
6. ◴[] No.26299005[source]
7. ianlevesque ◴[] No.26299011[source]
Try https://www.messenger.com/desktop
replies(2): >>26300943 #>>26304761 #
8. pcr910303 ◴[] No.26299267[source]
> I think we'll see this happen with Facebook Messenger. Both the apps and the website have become slow and painful to use and get worse every month. I think we'll start to see engagement numbers dropping because of this.

In fact, I think the iOS app for FB Messenger did get a redesign due to problems and it’s rewritten from scratch? I remember being pleasantly surprised after the big update… It became lightweight, integrates well with iOS and supports platform features.

On the other hand, the desktop app or the website is a shitshow :-(

replies(1): >>26302704 #
9. AshWolfy ◴[] No.26299314[source]
Does it? Just because something is good or successful doesnt mean it was made well. Thats why we are seeing stories of crunch, failures of management compensated by extreme overwork.
replies(1): >>26299362 #
10. rsj_hn ◴[] No.26299362{3}[source]
Now you are moving the goalposts to a philosophical discussion as to what types of business you like or don't like, and I'm not sure any progress can be made with that approach. Some will share your values, others will find them repugnant, and in any case it doesn't have much bearing on rockstar load times.
replies(1): >>26311382 #
11. karmasimida ◴[] No.26299384[source]
> I can fully believe the load times will result in fewer users and lower expenditure.

Is GTA online still attracts new users in droves? I doubt.

If the old users live with the loading time for years, they are likely to continue living with it. It would be nice if Rockstar fixes it, but I doubt it would be anything except a PR win.

replies(2): >>26304611 #>>26384217 #
12. tw04 ◴[] No.26299942[source]
You have just described why I laugh anytime someone complains that government is inefficient. ANY organization of sufficient size is "inefficient" because what a large organization optimizes for (for reasons I cannot explain) cannot align with what that organization's customers want optimized.
replies(4): >>26300873 #>>26301249 #>>26311226 #>>26313510 #
13. toyg ◴[] No.26300873[source]
> for reasons I cannot explain

Any sufficiently large institution, over time, will prioritise self-preservation over achieving their core mission. This is sociology 101. Once a company has enough users to make it hard or impossible to measure immediate performance, self-preservation is achieved with internal manoeuvering and selling to execs.

replies(1): >>26301047 #
14. lethologica ◴[] No.26300943{3}[source]
[tinfoil hat] A part of me believes that this was their intention in slowing down their browser site to the extent it has (and there is absolutely zero reason as to why it should load as slow as it does) in order to drive downloads for this application. [/tinfoil hat]

It's funny. That page says "A simple app that lets you text, video chat and stay close to people you care about." If it's so simple, then why does the browser version of the site take forever to load?

replies(1): >>26302797 #
15. jack_riminton ◴[] No.26301047{3}[source]
Whats the remedy? apart from reducing the size of the organisation
replies(6): >>26301334 #>>26301388 #>>26301392 #>>26301669 #>>26305078 #>>26305980 #
16. buran77 ◴[] No.26301249[source]
With the added difference that governments also have to be far more procedural by virtue of the way they are set up. Regardless of size they are accountable and responsible to a far higher degree in the eyes of the population they represent so there is a legitimate reason to be "slow".

In games the added reason to be slow is that game code is by definition some of the least mission critical code one could find (competes with 90% of the internet web code). Your Linux or Windows code might run a hospital's infrastructure or a rover on another planet. A game on the other hand can launch with bugs the size of your windshield, and can stay like that forever as long as people still pay. And people will pay because games are not unlike a drug for many people.

As such most game coding teams and coders are "trained" to cut every corner and skimp on every precaution. They're not needed beyond a very low baseline as far as software is concerned.

Look at the amount of bugs or cheats incredibly popular games like GTA or CoD have. These are billion dollar a year franchises that leave all this crap on the table despite all the money they make. They have all the resources needed, it's a conscious call to proceed like this, to hire teams that will never be qualified enough to deliver a high quality product and will be encouraged to cut corners on top of that.

Source: a long time ago I worked for a major game developer in a senior management role (unrelated to the dev activity) and left after feeling like "facepalm" for too long in every single SM meeting.

replies(1): >>26307251 #
17. ayewo ◴[] No.26301334{4}[source]
There was a fantastic discussion some years ago on ways to design an organization to minimize the tendency to drift towards self-preservation instead of remaining customer-focused.

The HN discussion[1] was started by an article that provided numbers that seemed to suggest that Wikipedia's spending was slowly spiraling out of control.

1: https://news.ycombinator.com/item?id=14287235

replies(1): >>26313817 #
18. pdimitar ◴[] No.26301388{4}[source]
Many people would say "more accountability" but I've seen that used successfully to deflect lightning strikes to innocent people who were then fired so... I'd like to know as well.
19. buran77 ◴[] No.26301392{4}[source]
Even if you take engineering you see that adding more links on a chain increases latency and demands more auxiliary circuitry. But at least in engineering you can design each part to do what you want it to do close to perfection. And it will scale better because you can build everything to spec and bin, which is why we automate a lot of tasks. With humans that can't happen. Human "binning" is a constantly moving target.

After tangentially working on this for a long time I'd say that the core issue is so deeply ingrained in human psyche that it may not even be a matter of education (starts early), let alone organization (starts happening when everything else is "set in stone"). There's no organizational structure that fits the types of activities we humans tend to do these days and that can deliver low latency, consistent results at scale. We mitigate issues in one area by accentuating them in others.

You can have one large flat structure but the load on the single coordinating circuit (the manager) will compromise quality. You can split the thing in multiple, individually coordinated units but the added layer of coordination and latency will compromise performance.

Maybe some form of general purpose but not quite full AI, something that combines human like intelligence and engineering like consistency, might be able to do what humans are supposed to but without the variability of humans (which is both good and bad).

replies(1): >>26301509 #
20. toyg ◴[] No.26301509{5}[source]
Introducing AI into work relations is how you turn every org into Uber or Amazon delivery: platforms where the worker has no real agency on his work, the apotheosis of alienation. I have no doubt that someone will try it (already we see it creeping in for hiring), I just think it will be Fundamentally Bad.
replies(1): >>26306146 #
21. donkeyd ◴[] No.26301669{4}[source]
I doubt there is a remedy and personally accept this as a fact of life.
22. stefs ◴[] No.26302435[source]
1. people experiencing this issue have already bought the game, so there's little incentive here.

2. we can be reasonably sure people will buy newer GTA installments regardless of whether this bug is fixed or not.

but:

3. if there's still money to be made from microtransactions this is a huge issue and would absolutely be worthwhile, imo.

23. Tsiklon ◴[] No.26302704[source]
The iOS app is really much more performant now than it was a couple of years ago. Significantly smaller, and quicker to start up
24. gocartStatue ◴[] No.26302797{4}[source]
Yeah, it used to work well in mobile browser, then only in desktop mode in mobile browser, then not at all. Now it only "kinda sorta" works in desktop browser on ridiculously overpowered PC. Gotta keep the walls tight in that private garden.
25. mywittyname ◴[] No.26303488[source]
I would think this would be one of the biggest revenue / developer_time changes in company history, considering how incredibly profitable online is.
26. thrwyoilarticle ◴[] No.26304611[source]
Before GTA online they entertained themselves in other ways. Eventually, they'll move on. The more friction there is to continue playing GTA online, the easier it is for there to be something to pull them away. Rockstar are now competing to be a use of someone's time, not for them to buy the game.
27. alkonaut ◴[] No.26304643[source]
> if a problem like this isn't fixed, it's because they've done the calculations and decided it's not worthwhile.

If it ever reached the point where it had to be an item in a priority list, it's already a failure. Some developer should have seen the quadratic behavior and fixed it. It's not the type of thing that should ever even be part of a prioritized backlog. It's a showstopper bug and it's visible for every developer.

28. saghm ◴[] No.26304761{3}[source]
Unfortunately I'm on Linux, and they don't seem to provide a Linux client. I did try out https://github.com/sindresorhus/caprine/, which worked really well, but I'm one of those weirdos who prefers websites to desktop apps.
29. tehjoker ◴[] No.26305078{4}[source]
It's not a perfect remedy, but you have to loop in the people affected by decisions as part of the decision making structure. That is, for example, customers and workers have to be part of the management structure.

This doesn't happen because it would reduce the power of top decision makers and potentially impact profits. e.g. a customer might ask for a chronologically ordered timeline on Facebook, but that would harsh engagement metrics, revenue, etc. If stuff like this did happen more often though, you'd get products and services that more often achieve their stated aims.

replies(1): >>26313572 #
30. SilasX ◴[] No.26305980{4}[source]
Competition/choice, which means that self-preservation requires that they care about efficiency. It obviously that wasn't enough here, but definitely tames some inefficiencies.
31. buran77 ◴[] No.26306146{6}[source]
> Introducing AI into work relations is

We don't really know what it is simply because we haven't introduced any "real" AI anywhere, let alone:

>> some form of general purpose but not quite full AI

Talking about efficiency as you scale up large organizations, it's inevitable that humans will introduce delays and variability in the work which cannot be eliminated because it's human nature, biologically and psychologically. Since humans can't change on a timescale that makes this discussion relevant, the only way for very large organizations (like large companies or governments) to operate just as efficiently as small ones is if they rely (quasi)exclusively on some AI that's as capable as top individuals at delivering results but with fewer of the drawbacks. Not only would it not operate as 100.000 distinct entities but as one or a few, it would also run consistently and predictably.

32. com2kid ◴[] No.26307251{3}[source]
Remember this story from a few days ago?

https://randomascii.wordpress.com/2021/02/16/arranging-invis...

I've seen plenty of interesting bugs, best I found personally was a compiler that was outputting files 1 byte at a time.

Games are riff with these sorts of bugs, but the volume of released games vs other types of software makes any sort of comparison unfair.

replies(1): >>26310093 #
33. buran77 ◴[] No.26310093{4}[source]
> but the volume of released games vs other types of software makes any sort of comparison unfair.

No comparison is entirely fair but I think your objection is unfounded. The quantity of games being released is irrelevant when we're talking about the quality of their code. Which is really bad for most big titles.

If anything the comparison was unfair in the other direction. Sure, an OS or browser have a lot of bugs. But if games and their code had a fraction of the scrutiny something like Windows gets you might just find out that 4 lines of code were written "by the book". It's something any honest game dev will confirm, game code is a stack of spaghetti code on top of more spaghetti code. The philosophy is that you can just go ahead with bad code because you can always fix it with a patch later on. Then you notice there's no widespread pushback from gamers (because unless it's an absolute bomb there won't be) and move on with the next round of "features", nobody has time for fixing bugs or combing the spaghetti.

One other problem is that eventually some people coming from the gaming industry will end up switching to other types of software development but will stick to the philosophy. I've done a lot of hiring over my career and of one thing I'm certain. Whenever someone came with most of their career in game development or most of the recent experience I asked for the CV to be put at the bottom of the stack. It was a lesson I learned the hard way.

Issues like the low criticality of game code, the "crunch" work style, the idea that you should just get it out there as quickly as possible, the lack of serious scrutiny into the matter, etc. all compound each other to create a coding (and coordination) style that's hard to shake off.

replies(1): >>26310236 #
34. com2kid ◴[] No.26310236{5}[source]
> No comparison is entirely fair but I think your objection is unfounded.

I should have clarified.

Doing an absolute numerical comparison in terms if "bad game code" to "bad B2C code" is unfair, because now days there is a lot more game dev code than B2C code (if we ignore the web, which is a rather lot of code to ignore I'll admit :) ).

All that said, most of the games on my phone run very well. Most of the games on my PC run very well. Do the massive over budget AAA titles have issues? Of course. Human organizations are really bad at creating large projects, we all know that social interaction overhead scales non-linearly with # of people, large projects get bogged down.

(Now throw in B2B software where the CTO makes the purchasing decisions and only people lower level in the company have to use it. All of a sudden enterprise software starts to suck as much as AAA games!)

> The philosophy is that you can just go ahead with bad code because you can always fix it with a patch later on.

Sure, for AAA games. But plenty of games that I play don't have those sorts of issues.

Meanwhile my Windows start menu only opens every other time I press the start menu button. This happened for years, then it was fixed for a bit, then a few patches later it started happening again.

The start menu on Windows should be the 2nd to last thing that breaks, right behind the mouse driver.

Without huge testing efforts, large software projects will collapse under its own weight. The type of software isn't material to the problem.

> One other problem is that eventually some people coming from the gaming industry will end up switching to other types of software development but will stick to the philosophy.

I've hired plenty of ex-game devs, I agree they have some holes in their skills. But I've also found out that if you sit them down they can write code that traditional software engineers couldn't manage. Spending too long in any one paradigm fixes ones mindset into that paradigm.

I also agree that the quality of code someone puts out is highly influenced by the environment they work in. But I've worked with plenty of ex game devs who take pride in writing correct code the first time around.

> Whenever someone came with most of their career in game development or most of the recent experience I asked for the CV to be put at the bottom of the stack.

That's a shame, because I wouldn't have accomplished some of the things I've done in my career if I hadn't hired a few game developers along the way! I've seen some stupid crazy stuff get pulled off, code that "shouldn't be able to exist", but did exist, and was rock solid reliable, all written by game developers.

People are people, and sure 9 out of 10 the person who bangs on kernels for a living is probably super careful about every line of code they write, but I'm not going to judge an entire field because the most bureaucratic outputs of that field are horrible.

That'd be like saying everyone who writes web apps is an incompetent developer because Jira is slow.

Or that all software developers are bad because, honestly, the majority of users only notice what we make when it breaks.

Well, we only hear about the small % of games that have huge problems, not the larger number of games that work just fine out of the box.

35. mbrodersen ◴[] No.26311226[source]
And I laugh every time people claim that it is only governments that can be inefficient. Most large commercial companies are inefficient and almost not functioning at all.
36. AshWolfy ◴[] No.26311382{4}[source]
I dont think anyone likes excessive loading times or crunch. You might argue about the importance of optimization or the necessity of crunch, but i dont think anyone can argue this isnt a failure of management and communication
37. imtringued ◴[] No.26313510[source]
Organizations are like spheres. Only a small part of the sphere has an exposed surface that is in contact with the outside world. As you grow the sphere most of the mass will be inside the sphere, not near the surface.
38. jack_riminton ◴[] No.26313572{5}[source]
I understand what you're saying. When I was working as a Product Manager, I had to try really hard to do things for the customer (after user testing etc.)

Normally the response from management was "but will that increase our profits?", to which my answer was "eventually, yes"

39. jack_riminton ◴[] No.26313817{5}[source]
That really is a fantastic quote and a fantastic thread!

What really stuck out is that Buffet said its the most important thing about Business and he didn't learn it in Business school. Same! I did an MBA and it was the biggest elephant in the room.

replies(1): >>26325243 #
40. brokenmachine ◴[] No.26325243{6}[source]
But he didn't specify how he arranged things to avoid the problem.
replies(1): >>26326773 #
41. jack_riminton ◴[] No.26326773{7}[source]
Fair point. The next poster explains how Amazon identifies and tries to mitigate it:

"if you’re not watchful, the process can become the thing. This can happen very easily in large organizations. The process becomes the proxy for the result you want. You stop looking at outcomes and just make sure you’re doing the process right"

replies(1): >>26335667 #
42. brokenmachine ◴[] No.26335667{8}[source]
Yeah, I thought that was very insightful.
43. ric2b ◴[] No.26384217[source]
I rarely play it and the load time is actually the main factor. If I have an hour to play a game waiting for GTA V to load unfortunately feels like a waste of time and a chore, so I play something else.