The scenario that concerns me is that Claude learns unpublished research ideas from me as we chat and code. Claude then suggests these same ideas to someone else, who legitimately believes this is now their work.
Clearly commercial accounts use AI to assist in developing intellectual product, and privacy is mandatory. The same can apply to individuals.
It has long been a problem in math research to distinguish between "no one has had this idea" and "one person has had this idea". This used to take months. With the internet and MathSciNet, ArXiv online it took many iterations of guessing keywords. Now, I've spent six months learning how to coax rare responses from AI. That's not everyone's use case.
What complicates this is AI's ability to generalize. My best paper, we imagined we were expressing in print what everyone was thinking, when we were in fact connecting the dots on an idea that was latent. This is an interesting paradox: People see you as most original when you're least original, but you're helping them think.
With the right prompts AI can also "connect the dots".