There is the question. See "Reflections on Trusting trust" (a classic paper). However in the end you cannot do everything you might want to and so you must trust someone else. Operating systems are common, audited by many, and used by enough that you can have high trust they work in general (but there are some not worthy of trust). Package managers tend to contain many packages that are not in common use and if the only one who audit them might be you so you better do it yourself for each release.
If you only use a package manager for libraries that you have high trust in then you don't need to worry - but there are so few projects you can have high trust in that manual management isn't a big deal. Meanwhile there are many many potentially useful packages that can save you a lot of effort if you use them - but you need to manually audit each because if you don't nobody will and that will bite you.