LLM Visualization

I have a related question I guess, it relates to how I can visualise the foundations of this beyond just a code implementation.

Where does this come from in abstract/math? Did we not have it before, or did we just not consider it an avenue to go into? Or is it just simply the idea of scraping the entirety of human knowledge was just not considered until someone said "well, we could just scrape everything?"

Were there recent breakthroughs from what we've understood about ML that have lead to this current explosion of research and pattern discovery and refinement?

From my understanding, the field of AI was all about knowledge representation. Researchers in the past handcrafted representations with expert knowledge but that only gets you so far. So instead, why not learn representations from data directly?

That's the current stage we're at and is the whole scraping the entirety of human knowledge thing. Compute has gotten good enough and data readily accessible to do all this, plus we have architectures like transformers that scale really nicely.

I think it’s two fold: the evolution of hardware (GPUs) that give us enough compute power, and the invention of novel algorithms (transformer) that can effectively consume and understand all these data.