So they trained LLM's on a bunch of junk and then notice that it got worse? I don't understand how that's a surprising, or even interesting result?
replies(3):
And while this result isn't extraordinary, it definitely creates knowledge and could close the gap to more interesting observations.