←back to thread

Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data

(epochai.org)
56 points trott | 2 comments | 18 Jun 24 02:04 UTC | HN request time: 0.421s | source
Show context
makapuf ◴[18 Jun 24 06:52 UTC] No.40714795[source]▶
Funny that it does not need that much data to train your average 20th century human genius. I'd say that if we are dreaming of the future of ai, learning and reasoning seems the greatest issue, not data. That said, the article title is about LLMs, so that's what will need changing I guess.
replies(3): >>40715430 #>>40715643 #>>40716666 #
jstanley ◴[18 Jun 24 08:45 UTC] No.40715430[source]▶
Humans aren't just text interfaces though. The majority of your input is not textual but is sights, sounds, feelings, etc., that LLMs don't (yet?) have access to.

Humans receive an enormous amount of training data in forms not currently available to LLMs.

If you locked baby Einstein in a room with the collected works of humanity and left him there for a lifetime, I doubt he'd have even learnt to read on his own.

replies(6): >>40715609 #>>40715647 #>>40715822 #>>40715950 #>>40716247 #>>40716485 #
1. mrweasel ◴[18 Jun 24 09:26 UTC] No.40715647[source]▶
We also have a funny way of applying solutions and lessons learn in one field to problems in completely unrelated areas. Given the statistical nature of LLMs I'm not convinced that they are able look across fields in the same way as a human brain, they lack creativity.

The greatest advantage you can have in life is a creative mind and I don't believe that is something that can be taught. It can be stomped out of you as a child, but it's not learnable.

replies(1): >>40716407 #
2. lassoiat ◴[18 Jun 24 11:19 UTC] No.40716407[source]▶
I have come to the point that it is not really fair to the LLM to statistically train it on human output and expect it to come up with something more than the average.

There will be much value in automating the tedious and the routine.

Of course, that doesn't make for a great science fiction story. We first have to placate all these science fiction fantasies and in the process we will automate the tedious and the routine as a side effect of trying to figure out how many AGI can dance on the head of a pin.

Then human creativity will just be worth all the more.