In my opinion, training models on user data without their real consent (real consent = e.g. the user must sign a contract or so, so he's definitely aware), should be considered a serious criminal offense.
replies(5):
And when talking specifically about AI, one could argue that learning from interactions is a common aspect of intelligence, so a casual user who do not understand details about LLMs would expect so anyways. Also, the fact that LLMs (and other neural networks) have distinct training and inference phases seems more like an implementation detail.