←back to thread

Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data

(epochai.org)
56 points trott | 1 comments | 18 Jun 24 02:04 UTC | HN request time: 0s | source
Show context
makapuf ◴[18 Jun 24 06:52 UTC] No.40714795[source]▶
Funny that it does not need that much data to train your average 20th century human genius. I'd say that if we are dreaming of the future of ai, learning and reasoning seems the greatest issue, not data. That said, the article title is about LLMs, so that's what will need changing I guess.
replies(3): >>40715430 #>>40715643 #>>40716666 #
jstanley ◴[18 Jun 24 08:45 UTC] No.40715430[source]▶
Humans aren't just text interfaces though. The majority of your input is not textual but is sights, sounds, feelings, etc., that LLMs don't (yet?) have access to.

Humans receive an enormous amount of training data in forms not currently available to LLMs.

If you locked baby Einstein in a room with the collected works of humanity and left him there for a lifetime, I doubt he'd have even learnt to read on his own.

replies(6): >>40715609 #>>40715647 #>>40715822 #>>40715950 #>>40716247 #>>40716485 #
trott ◴[18 Jun 24 09:56 UTC] No.40715822[source]▶
The stream of data from vision does NOT explain why humans learn 1000x faster: Children who lost their sight early on, can grow up to be intelligent. They can learn English, for example. They don't need to hear 200B words, like GPT-3.
replies(3): >>40716628 #>>40716999 #>>40720531 #
1. LoganDark ◴[18 Jun 24 12:27 UTC] No.40716999[source]▶
Humans use bottom-up reinforcement learning, but nearly all LLMs use gradient descent. Not only are those completely different directions (bottom-up as in humans versus top-down as in gradient descent) with completely different emergent behavior, but minimizing loss is not in the reward function of a human, even if schools like to think it makes for an effective education. (I'd argue it doesn't.)