I'd rather be able to update my dependencies automatically with a few commands instead of manually vendor all my dependencies, keeping up to date is really important for security. I get that game developers who only ever work on building single player games might have different opinions on "package managers", but they are in a very small niche.
One of the worst things working at companies shipping C++ was the myriad of meta-build systems that all tries to do dependency management as a part of the build system without having a separate concept of what a "package manager" is, this is truly the worst of both worlds, where people are happy to add dependencies, never update them, and never share code between projects and departments. I do not wish that way of working on my worst enemies.
Whatever problems package management brings is such a better problem to have than not having a package manager. That said I think everyone can get better at being more discriminatory of what they add to their project.