←back to thread

Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data

(epochai.org)
56 points trott | 2 comments | 18 Jun 24 02:04 UTC | HN request time: 0s | source
Show context
ofrzeta ◴[18 Jun 24 04:35 UTC] No.40714106[source]▶
"If trends continue, language models will fully utilize this stock between 2026 and 2032" - that will require data centers with their own nuclear reactors (or other power plants) as hinted at by Marc Zuckerberg?
replies(2): >>40714248 #>>40714295 #
trott ◴[18 Jun 24 05:09 UTC] No.40714248[source]▶
If you take Llama-3-400B, and 30x its data (hitting the data ceiling, AFAICT), 30x its size to match, and the hardware improves by, say, 3x, then you'll use up about a year's worth of energy from a typical nuclear power plant.
replies(3): >>40714269 #>>40714294 #>>40714809 #
spiralk ◴[18 Jun 24 05:21 UTC] No.40714294[source]▶
If its for training a new foundation model it is not that bad. It's still only a fraction of the energy compared to many human industries. I did rough math some time ago and found that that training llama-3-70B used the equivalent energy to 1/30 of a full loaded container ship going from China to the US. Even scaled up 100x and trained 10x longer, its seems like the energy consumption is relatively small compared to other industries. The fact that people are considering nuclear power for AI training is an advantage not a downside, imo. It should have a much lower CO2 footprint.
replies(1): >>40715462 #
adrianN ◴[18 Jun 24 08:52 UTC] No.40715462{3}[source]▶
You always have to compare the cost to the value it generates. A year of power from a nuclear plant might be used in more productive ways.
replies(2): >>40715605 #>>40719559 #
1. ofrzeta ◴[18 Jun 24 09:17 UTC] No.40715605{4}[source]▶
Yes, and consider that with the current hype around AI and enough (venture) capital there will be several corporations competing for the best AI and suddenly we are at several "nuclear plants" or equivalent energy "consumption".
replies(1): >>40719695 #
2. spiralk ◴[18 Jun 24 16:36 UTC] No.40719695[source]▶
I don't see increasing demand for nuclear power as a disadvantage. We have nuclear material that can last humanity 1000s of years at least. The CO2 footprint is an issue but nuclear is much better than others. Personally, I think it's better we utilize more energy and discover new breakthroughs while society is relatively stable and functioning, because there's no guarantee that it will last. Population collapse seems imminent in more educated societies, even China and India are trending this way now. Without some level of AI assistance, humanity would likely lose a great deal of productive output.

Also, if this path to AGI does not work out, its not as though the nuclear reactors will be wasted. People will find something else to do with the energy.