Even if you want the most cynical approach - networking is an essential aspect of career development. If you stick to just "job description during work hours" you won't get far.
Building a cohesive team involves sharing interactions like this. If you truly value your privacy, I suggest you come up with an alternative thing to discuss: sports, weather, funny stories, etc.
I've certainly had weekends where I very much did not want to share what i did, so I deflected in a way that kept the conversation going.
If HR is asking about where you were at 3:15pm on Saturday, and who you were with, and whether you were using work resources at that time, I would start to worry. But a very large and generic, "so what did you do this weekend" - where you are in the driver's seat, that's not the same thing.
When I look back at previous jobs, my strongest regrets are not building stronger interpersonal relationships with my coworkers. I've since rectified that.
Think of it this way, if you must: coworker relationships are owned by the company - personal relationships are owned by the employees. When you become friends with a coworker, you acquire an asset that will go with you when you leave the company. Much like a new skill, this asset will allow you to get higher pay somewhere else.