Yeah, I've been doing this a lot lately. It's kind of disappointing how easily the recommendations shift once you click on a video that's sketchy.
In my case, I once made the bad decision to watch a Jordan Peterson video. By itself it wasn't crazy, he was mildly provocative, a little paternalistic, not my cup of tea. But geez, the trash that then ended up appearing in the side bar after that was awful. I find that if I watch some "bad" videos, I have to spend a fair amount of time culling shit out of the subsequent recommendations.
Another way to keep your feed from being polluted is simply to create "garbage user" that you toggle to if you want to view potentially garbage content.
What I really wish youtube did was to allow users to filter out videos that contain keywords we choose, like twitter's somewhat effective "muted words" list.