LLMs can get "brain rot"

> continual exposure to junk web text induces lasting cognitive decline in large language models (LLMs).

TLDR: If your data set is junk, your trained model/weights will probably be junk too.