OpenAI, Google and Anthropic are struggling to build more advanced AI

Every negative headline I see about AI hitting a wall or being over-hyped makes me think of the early 2000's with that new thing the 'internet' (yes, I know the internet is a lot older than that). There is little doubt in my mind that ten years from now nearly every aspect of life will be deeply connected to AI just like the internet took over everything in the late 90's and early 2000's and is now deeply connected to everything now. I'd even hazard to say that AI could be more impactful.

And, as I've noted a couple of times in this thread, how many times have we heard that Moore's law is dead and compute has hit a wall?

AI can be thought of as the 2nd stage of the creature that we call the Internet. The 1st stage, that we are so familiar with, is about gathering knowledge into a giant and somewhat organized library. This library has books on every subject imaginable, but its scale is so vast that no living human today can grasp it. This is why the originally connected network has started falling apart. Once this I becomes AI, all the books in the library will be melted together into one coherent picture. Once again, anyone anywhere on Earth will be able to access all the knowledge and our Babylon will stay for a little longer.

It's strange to me that's your takeaway. The reason that the internet was overhyped in the 2000's is because it was and also heavily overvalued. It took a massive correction and seriously disruptive bubble burst to break the delusion and move on to something more sustainable.

Even if you're right (you're not) whatever "AI" looks like in 20+ years will have virtually nothing in common with these stupid statistical word generators.

I disagree that it was over hyped. It has transformed our society so much that I would argue it was vastly under-hyped. Sure, there were a lot of silly companies that sprang up and went away because they weren't sound, but so much of the modern economy is based on the internet that it is hard to say any business isn't somehow internet related today. You would be hard pressed to find any business anywhere that doesn't at least have a social media account. If 2000 was over-hyping things I just don't see it.

pets.com was valued at $400 million based almost completely on its domain name. That's the classic example. People were throwing buckets of money at any .com that resolved to a site and almost all of it failed. I'm not sure how that doesn't meet the definition of over-hyped. It feels very similar to now. Not even to mention - the web largely doesn't consist of .com sites anymore, it's mostly a few centralized sites and apps.

Well according to Nvidia you can just ignore Moore's law and start requiring people to install multi kilowatt outlets just for their cards. Who needs efficiency amirite?

There were no smartphones in 2000, so the Web was overvalued at that point in time... until we all started carrying the Web in our pockets in the form of a portable rectangle.

Given that this is the case, why can't this be analogously true of “AI” as well? There's plenty of reason to believe that we're hitting a wall, such that, to progress further, said wall must be overcome by means of one or more breakthroughs.

'smartphones' needed a reason to exist, the internet provided that. I doubt we would have had them without it. AI will drive whole new product categories that didn't exist that will then transform our society even more.

I'm not an apple fan (as I type on a mac that I am forced to use) but I gotta applaud their push for power efficiency. NVIDIA actually -does- have a few cards they make that really improve power efficiency but then they generally hamstring them with a lack of memory. NVIDIA is really good at making their high-end cards the only viable choice but I think that will backfire on them as people like me, that value quiet, cool and efficient over 25% faster inference start taking any viable alternative that comes out.

That's funny, because to me these headlines about how deep learning is over-hyped and hitting the wall remind me of headlines from ten years ago about how... deep learning is over-hyped and hitting the wall.

That was before people could generate animation and music.

Wasn't that mostly from public markets which never invested in tech before?

There is a graveyard of hardware companies from the 70s, 80s, and 90s.

Memory is king.

Anything that has more memory and adequate compute will win the coming AI wars.

At the rate at which power consumption is growing now that the shortage of current gen cards has started to work itself out people are realizing they need a fleet of nuclear reactors to keep the data centers running. This is not something that's getting fix with the coming generation, if anything it's worse.

AI was overhyped in the 1950s with the perceptron. Machine learning advances in fits and starts. As soon as it looks like it’s out of steam something novel comes out. Circa 2010 all the effort was on perfecting SVMs to the point where 1% point improvement on a computer vision task was a PhD thesis and the like then all of a sudden AlexNet made neural nets look feasible and the game changed overnight.

I think everyone knew the internet would change everything and thus be very valuable. At the time the web was the primary manifestation of the internet. Domain names felt like an oil rush to carve up the internet. But it was actually a rush to carve up the web, and no one realized yet that things like Google search and app stores would make domain names far less valuable over time.

Plus, they're "struggling"? Of course they are! It's cutting edge, and it's hard. If they weren't struggling, it would have been done long ago.

There are a lot of comparisons that could be drawn: web 3.0, the internet, the dot com bubble, etc. but I think the most appropriate comparison would be to... AI in the past. No one doubts that there was a lot of value coming from that research. In fact a lot of it is incorperated in our every day life. But it didn't live up to its hype. I suspect the same will be true for this wave of AI (and perhaps an associated AI winter).

a lot of which were founded on the promises of AI: symbolics, thinking machines corporation

you do realize all of the worlds most valuable companies are either built on/for internet or heavily leverage it

My recollection of AI in the past is that it was nothing like this.

If you look at the Wikipedia article 'History of artificial intelligence' for now it has 'AI boom' and '2004 Nobel Prizes' but everything earlier is kind of meh.

I remember sitting down with pen and paper to try to write a ChatGPT type chatbot 44 years ago and of course totally failing to get anywhere, but I've followed the goings on since and this is the first time this stuff is working well.

And essays for homework assignments that would get a decent grade, art for the headings of blog posts, rewrites of an email to make it sound more professional, summaries of long documents, or just generally to create something that gives the semblance that you did a lot of work while actually having done very little work.

And yes of course hallucinations are a huge problem for most of these use cases, but they aren't stopping people from using them anyway. We have a new misinformation problem and it has no agenda. It's basically just white noise.

So my money is also on this changing the world dramatically, just not in the in uniformly positive way that the hype said it will.