I wonder how many anti-torrent groups are doing this covertly.
Definitely a few. Media companies often send out infringement notices to ISPs to be forwarded to the user and I would guess this is how they get those IPs
No tracker necessary, you can just use DHT:
https://iknowwhatyoudownload.com/huh, weirdly it has stuff I did download and stuff I didn't download within minutes of each other ... should I be worried?
Stop using DHT and/or public trackers and you will be safe. They scan public trackers and the DHT network.