←back to thread

OpenAI, Google and Anthropic are struggling to build more advanced AI

(www.bloomberg.com)
625 points lukebennett | 1 comments | 13 Nov 24 13:28 UTC | HN request time: 0s | source
Show context
pluc ◴[14 Nov 24 18:23 UTC] No.42139375[source]▶
They've simply run out of data to use to fabricate legitimate-looking guesses. They can't create anything that doesn't already exist.
replies(7): >>42139490 #>>42140441 #>>42141114 #>>42141125 #>>42141590 #>>42141888 #>>42149715 #
whazor ◴[14 Nov 24 19:53 UTC] No.42140441[source]▶
But a LLM can certainly make up a lot information that never existed before.
replies(2): >>42141540 #>>42142063 #
bob1029 ◴[14 Nov 24 21:42 UTC] No.42141540[source]▶
I strongly believe this gets into an information theoretical constraint akin to why perpetual motion machines don't work.

In theory, yes you could generate an unlimited amount of data for the models, but how much of it is unique or valuable information? If you were to compress all this generated training data using a really good algorithm, how much actual information remains?

replies(3): >>42141792 #>>42141948 #>>42181780 #
1. rocho ◴[19 Nov 24 10:07 UTC] No.42181780[source]▶
That's correct. I saw a paper recently that showed how LLMs performance collapses when they are trained on synthetic data.