Shouldn't the right's extend forward and simply require the LLM code to be deleted?
Those kinds of cases, although they do happen, are exceptional. In a typical output that doesn't not line-for-line resemble a single training input, it is considered a new, but non-copyrightable work.
You should be careful about speaking in absolute terms when talking about copyright.
There is nothing that prevents multiple people from owning copyright to identical works. This is also why copyright infringement is such a mess to litigate.
I'd also be interested in knowing why you think code generated by LLMs can't be copyrighted. That's quite a statement.
There's also the problem with copyright law and different jurisdictions.