Making games in Go: 3 months without LLMs vs. 3 days with LLMs

Game developers will try anything before they actually write automated tests for their games.

When you tweak game mechanics several times every day, keeping the tests useful is a large task. Basics can be tested. Map integrity can be tested. Most "normal UX" is hard to test, and even main functional tests tend to drift. (Source: a short involvement in actual gamedev recently.)

One can still write unit tests. I have been told from a couple different game devs that it's more because of release deadlines, and the cost of a bug is usually pretty small.

There are some game systems that lend themselves to unit testing, like say map generation to ensure that the expected landmarks are placed reasonably, or rooms are connected, or whatever. But most game interactions are just not easily "unit testable" since they happen across frames (eg over time). How would you unit test an enemy that spawns, moves towards the player, and attacks?

I'm sure you could conjure up any number of ways to do that, but they won't be trivial, and maintaining those tests while you iterate will only slow you down. And what's the point? Even if the unit-move-and-attack test passes, it's not going to tell you if it looks good, or if it's fun.

Ultimately you just have to play the game, constantly, to make sure the interactions are fun and working as you expect.

> How would you unit test an enemy that spawns, moves towards the player, and attacks?

You use a second enemy that spawns, moves towards the "enemy", and attacks.

> How would you unit test an enemy that spawns, moves towards the player, and attacks?

You can easily write a 'simulation' version of your event loop and dependency inject that. Once time can be simulated, any deterministic interaction can be unit tested.

Others would quibble that those are integration tests, "UI" tests, or other higher-level tests, etc.

I've heard the same excuses from ML engineers before introducing tests there, embedded engineers, robotics engineers, systems engineers, everyone has a reason.

The real reason? It's because writing tests is a different skill and they don't actually know how to do it.

The problem with tests for games is that a lot of game code is in constant flux. A test suite introduces a not insignificant amount of rigidity to your codebase. Pivot a few concepts and you have dozens of tests to fix - or just invalidate entirely. Very basic stuff that won't ever change can be tested - like whether the renderer is working properly - but that's never where the difficulty in game dev lies and its the stuff usually handled by a third party - library or engine.

Oh that's crap. I've been a software engineer for over 30 years. I love tests - I preach testing at my current place of work. I've also worked in games for about a decade. Testing in games is... not useless, but very much less useful than it is in general software engineering.

As a counter example I found this video essay about fixing a factorio bug fascinating. My main takeaway, I need better introspection hooks. I am not really programmer and I never really thought automated testing of user interactive parts was possible.

https://www.youtube.com/watch?v=AmliviVGX8Q (kovarex - Factorio lets fix video #1)

League of Legends does a lot of automated testing for their gameplay logic https://technology.riotgames.com/news/automated-testing-leag...

A lot of games aren't deterministic within a scope of reasonable test coverage.

Set the same seed for the test?

> and the cost of a bug is usually pretty small.

Like letting speed runners skip half your game. :)

Which is all the same as what unit test was originally defined as.

You're right that "unit test" has taken on another, rather bizarre definition in the intervening years that doesn't reflect any kind of tests anyone actually writes in the real world, save where they are trying to write "unit tests" specifically to please the bizarre definition, but anyone concerned about definitional purity enough to quibble about it will use the original definition anyway...

It would depend on how things are architected, but you could definitely test the components of your example in isolation (e.g. spawn test, get the movement vector in response to an enemy within a certain proximity, test that the state is set to attacking, whatever that looks like). I don't disagree that it's a hard problem. I run into similar issues with systems that use ML as some part of their core, and I've never come up with a satisfying solution. My strategy these days is to test the things that it makes sense to test, and accept that for some things (especially dynamic behavior of the system) you just have to use it and test it that way.

> The problem with tests for games is that a lot of game code is in constant flux. A test suite introduces a not insignificant amount of rigidity to your codebase. Pivot a few concepts and you have dozens of tests to fix - or just invalidate entirely.

Sounds very much like the description of a big ball of mud.

An interesting gamedev video I saw recently basically boiled down to: "Build systems, not games." It was aimed at indie devs to help with the issue of always chasing new projects and making code that's modular enough to be able to reuse it.

But taking a step back, that very much feels like it should apply to entire games, where you should have boundaries between the components and so that the scope of any such pivot is managed well enough not to tank your velocity.

Other than that, it'd be just the regular growing pains of TDD or even just needing to manage good test coverage - saying that tests will eventually need changes isn't the best argument against them in webdev, nor should it be anywhere else.

> Sounds very much like the description of a big ball of mud.

I mean, yeah, kinda.

For any given object in the game world, it's funnest for that object to be able to interact with as many other objects as possible in as many ways as possible. A game object's handles for interaction need to be globally available and can't impose many invariants—especially if you don't want level designers to have to be constantly re-architecting the engine code to punch new holes for themselves in the API. Thus, a lot of the logic in a given level tends to live inside the callback hooks of level objects, and tends to depend on the state of the rest of the level for correctness.

Modularity is a property of high cohesion and low coupling, which are themselves only possible when you can pin down your design and hide information behind abstraction boundaries. But games are a flexible and dynamic enough field that engines have to basically let designers do whatever they want, whenever they want in order for the engine to be able to build arbitrary games. So game design is naturally a highly-coupled, incohesive problem space that is poorly suited to unit testing.

> So game design is naturally a highly-coupled, incohesive problem space that is poorly suited to unit testing.

Poorly suited? Perhaps, but so are certain web system architectures as well, neither is impossible to test.

I think Factorio is an example that it can be done if you care about it... it's just that most studios shipping games don't.

https://www.factorio.com/blog/post/fff-438

https://www.factorio.com/blog/post/fff-366

Of course, in their case it can actually be justified, because the game itself is very dependent on the logic working correctly, rather than your typical FPS game slop that just needs to look good.

Yeah I suspect Factorio's "complex game logic + simple(ish) 2d engine + minimal team structure" situation meant that the usual tradeoffs didn't apply. It's really cool that they pulled it off, though—I can't imagine it was easy, even then.

I have no experience in the gamedev industry. But based on the number of bugs I see in games, plus the size and quantity of post-release patches, maybe your perspective here is because you're not trying to hit a level of reliability at launch that would justify having more tests?

I think that's indicative of not having enough QA. QA IS effective in the context of games for discovering bugs.