However, witnessing the rapid evolution of AI with just a few hundred GPUs, enough data, and power, I no longer wonder what a billion years of feedback loops and randomness can achieve.
A single training session will iterate more than the number of generations of all birds.
But that's not the right analogy. Evolution happens at the individual level, and even to some extent at the individual gamete level. So it's actually every single fecundated bird egg that ever existed, and even every single spermatozoon and egg cell every time two birds mated. Not to mention every division of every bacterial cell in every bird gut, since microflora are a key part of the organism too.
And even this is an undercount, since the DNA and gene expression of an individual actually changes during its lifetime, and those changes can be passed down to offspring through various mechanisms. So there is a constant process of evolution that even all cells inside a living organism go through, that we're still trying to fully understand.
That statements need a whole lot of backing. It contradicts the Central dogma of molecular biology [0]. The idea that an organism can pass on to its offspring physical characteristics that the parent organism acquired through use or disuse during its lifetime is called Lamarckism which was disproven more than a hundred years ago.
[0] https://en.wikipedia.org/wiki/Central_dogma_of_molecular_bio...
For an extremely simple example, if a female organism is exposed to significant doses of radiation, the DNA inside the egg cells it was born with will have a high chance of suffering mutations. If one of those eggs remains viable and later gets fertilized, the DNA of the offspring produced from it will be significantly different from the DNA that of offspring born before the radiation exposure.
There are other more subtle and more common processes, though. The most surprising one is that DNA methylation, a process in which environmental stimuli encountered in a living sexually mature individual affect gene expression, has been proven to be transmissible to offspring across several generations in certain species. This includes stimuli such as prolonged exposure to heat or cold, long-term starvation, and others. This is indeed a form of Lamarckism that has actually been proven to happen, though of course it involves much subtler changes than some of what Lamarck envisioned.
I don't think that's the right way to think about it. Genetic variation originates at an individual level, but "evolution" (differentiated survival of genetic variants) mostly happens at population level after genetic changes have spread among the population. In particular this is the case with "punctuated equilibrium" that has been observed in the fossil record (long periods of stability, interspersed with short periods of rapid change).
What happens is that many genetic variations are subtle and don't have an immediate survival benefit to the individual, so will just accumulate in the population as a whole as they are spread by breeding. Once in a while some environmental change occurs (famine, drought, disease, new predator, etc/etc) that may make a set of accumulated genetic changes, previously benign, now become more important to survival. With multiple sub-populations of the same species that have genetically drifted apart over multiple generations, some will now become more successful than others in this new (changed environment) evolutionary landscape.
Ultimately this sub-population genetic drift may lead to inability of these sub-populations (e.g. plains elephants vs forest elephants) to interbreed, and then no-going-back speciation has occurred, and further drift is guaranteed (due to no interbreeding to merge genetic changes).
I should have expressed myself more carefully. What I am thinking of is that each individual is essentially one training step, where the current model is confronted with the training data (in this case, the entire environment, including other members of the population). Or even multiple moments over the life of the individual might be considered distinct training steps, where the model is adjusted in minute ways (epigenetics) based on certain events.
Of course, this makes less sense for species that are highly communal, such as bees, ants, or termites, where the fitness of an entire population is highly interconnected, and even severe maladaptations in an individual can nevertheless be an improvement to the fitness of the overall population (e.g. sterility in the vast majority of individual bees and ants is not a detriment, when it is a fatal flaw in almost all other animals).
LLM training is training an organism to "survive" in an environment consisting of languages/lexicons.
Not getting eaten by a tiger is equivalent to being able to produce a semantically and logically correct sentence.