To make it respect user privacy I would use this data for training preference models, and those preference models used to finetune the base model. So the base model never sees particular user data, instead it learns to spot good and bad approaches from feedback experience. It might be also an answer to "who would write new things online if AI can just replicate it?" - the experience of human-AI work can be recycled directly through the AI model. Maybe it will speed up progress, amplifying both exploration of problems and exploitation of good ideas.
Considering OpenAI has 700M users, and worldwide there are probably over 1B users, they generate probably over 1 trillion tokens per day. Those collect in 2 places - in chat logs, for new models, and in human brains. We ingest a trillion AI tokens a day, changing how we think and work.