No clicks, no content: The unsustainable future of AI search

The argument seems flawed to me: by "killing the web", they refer to the example of a company adding SEO'd information to their website to lure in traffic from web searches.

However, me personally, I don't want to be lured into some web store when I'm looking for some vaguely related information. Luckily, there's tons of information on the web provided not by commercial entities but by volunteers: wikipedia, forum users (e.g. StackOverflow), blogs. (Sure, some people run blogs as a source of income, but I think that's a small percentage of all bloggers.)

Have you ever looked for a specific recipe just to end up on someone's cooking website where they first tell your their life story before - after scrolling for a half a day - you'll finally find what you've actually come there for (the recipe!) at the bottom of their page? Well, if that was gone, I'd say good riddance!

"But you don't get it", you might interject, "it's not that the boilerplate will disappear in the future, the whole goddamn blog page will disappear, including the recipe you're looking for." Yeah, I get it, sure. But I also have an answer for that: "oh, well" (ymmv).

My point is, I don't mind if less commercial stuff is going to be sustainable in a future version of the web. I'm old enough to have experience the geocities version of the early web that consisted of enthusiasts being online not for commercial interests but for fun. It was less polished and less professional, for sure, but less interesting? I don't think so.

> Luckily, there's tons of information on the web provided not by commercial entities but by volunteers

The question is: is there content which is useful, but not provided by volunteers? We see more and more content behind paywalls, and it is a loss for many people who can't pay, because they won't be able to access the same content for free supported by ads.

So the result is poor people are going to lose access to certain contents, while well to do people will still have access.

Sadly, i think the future is more crap. It is simply too easy to generate text you think people might want to read (even if it is not).

On the one hand, I agree with what you've said.

On the other, I think it's unlikely the fun old geocities era comes back.

We'll probably get stuff that looks like it, but it's hawking nationalist revisionist propaganda instead of occult shapeshifting magic lifted from Sabine Baring-Gould, and a thousand Temple OS-inspired clones instead of python.

The same amount of quality content might still exist even when the advantages to providing it dry up further.... But good luck finding in an ever growing sea of slop.

We might see a resurgence in curated content but I have my worries. Google gets worse and worse but also traditional curated sites have started simply repost what's trending on Twitter.

AI will kill the volunteers run websites and blogs faster then it will kill corporate ones. It will kill free information first. It will basically finish the process google search engine started when it started to require seo to find stuff.

People will have less or no motivation to create them, because well, why would they? It will be just a food for AI of some corporation.

And more importantly, people won't be finding and joining communities that produce the websites like stack overflow.

It was nice while it lasted, but likely it will be something that existed only for one generation.

Another commonly ignored group are those that publish information because they want others to know about that information. Even in a worst-case situation where websites are only crawled by bots and get no human visitors at all anymore, government and company websites won't disappear. Any tourist board worth their salt will increase the amount of content they publish compared to current levels. The local conspiracy nut will have an incentive to continue their blog, as will any university researcher. Press releases about new scientific discoveries will continue. Your personal blog might die, but lots of current "linkedin influencers" will start blogs to ensure LLMs think positively about their skills.

And that's assuming a world where people only ask LLMs and don't care about the provenance or trustworthiness of information at all. Which seems unlikely, even conspiracy nuts have some sources they trust more than others. The web will be fine. It will change drastically with the death of click-based advertising, but I don't see a future where it disappears

People create content for free but not to be sucked into some machine that funds a $T company who mixes it up and spits it out without attribution. You want to share info for the sake of it but you also care deeply about how it’s presented and would like people to come read it on your site. So all that volunteer stuff will be killed along with all the recipe sites because those people aren’t going to put in the time and effort without some financial return (the pages of ads we detest but put up with).

In the short term it will feel liberating; in the long term it will kill the web.

I doubt search engines are primarily what helped most blogs and sites grow. It's forums/word of mouth/social media

Often, that off-topic content was garbage anyway. Not really deep or helpful or expert in any way. Not always, but often.

As the world always has been. There is no human right that we must have free content supported by ads. And ad supported content has tons of its own issues.

The future is already here, and it's already crap. The web has been concentrated and filled with spam for a decade if not more.

The recipe thing is because recipes are not copyrightable and many duplicates exist around the internet. The padded length makes the content unique / not penalized as a duplicate and as a secondary benefit avoids penalties for short content.

Not saying this is good, but it’s the reason behind it.

Yes. I'm saying we're about to see another few orders of magnitude of it.

Recently i've noticed google is even less effective than normal because it's turning up ad-filled pages that vaguely relate to the query but are clearly entirely generated by an LLM that also doesn't know what you're looking for.

So, we went from 95% crap to 100% crap over the course of about two years. There is obviously still stuff out there worth finding, but I can't imagine LLMs are going to get us there.

But AI is also going to kill some of your positive examples. Stack Overflow for example is in a steep decline, only a small fraction of questions are posted today compared to the peak. And the effects are more than financial, so even non-profit examples like forums would be hit.

If new people don't discover your site with useful user-created content, they won't contribute to it. You're also cutting off the pipeline for recruiting new users to your forum or Q&A site.

Stack overflow was a shit show way before LLMs became popular.

I have no intention to stop writing on my blog for AI reasons and I don't even see why I should. I suspect a majority of people who post here are the same.

Stuck overflow may not be the greatest example. I have switch to using GitHub discussions and Discord on the "where to get help for my projects" side of things. I ignore SO when it comes to support. Lots of other open source projects doing similar.

This trend was happening before LLMs entered the arena.

Discord is just absolutely worthless for this. Any question that gets asked gets buried in days if not hours. It pretty much guarantees the same basic garbage gets repeated over and over and over forever. Basically the exact opposite of stack overflow.

Inevitably too you'll get someone scolding you to "check the pins" which you then do and get introduced to that hellish nightmare.

Discord is great for chatting with your friends, gaming, etc. but man it's a horrible knowledge repository.

There are question/answer channels, not everything is chat on Discord

> horrible knowledge repository

I don't disagree, but that does not change the fact that people have moved from sites like SO to Discord for this purpose.

There are Q&A channels, so not everything is chat, but Discord search is abysmal

Slack is another place where former SO content / answers are happening. Discourse too. The tl;dr is that it has become more fragmented, for better or worse

SO has a related problem to Reddit. Some mods high on their status and power

The answer to that question is absolutely yes. Investigative journalism, which is some of the most useful content in existence can not sustainably be provided by volunteers.

> many people who can't pay

Everybody is already paying for Spotify and for Netflix. They can pay for mass syndication of textual content. But it needs to be like Spotify or YouTube, where everything and anything goes. Poor people always had access to read newspapers.

That brings up the unstated part of the original comment. There's this obsession with ossifying the Internet because we're so afraid of losing something like SO.

The people who made SO are not going anywhere, there will always be a SO, a wikipedia, a search engine. Let it evolve to the next thing.

> the geocities version of the early web that consisted of enthusiasts being online not for commercial interests but for fun

LLMs are making many of the enthusiasts who were online just "for fun" feel sick for contributing to their training set.

> someone's cooking website where they first tell your their life story before

I haven't seen a recipe page without a "Jump to Recipe" page button in forever.

Surely part of that is because most tech related questions have already been asked and answered on SO. I'd say a decline in new question is stack overflow working as designed. A large part of what makes SO so good is the searchability of old questions. There will always be new questions to ask, as new technologies confuse Devs in new ways, but to expect new questions to be asked at the same rate as peak is to misunderstand what SO is at it's core.

But that's just like, my opinion, dude.

Google making blogs impossible to find if you dont do SEO already started that process. There is less of this kind of activity then it used to be. Some people who write mostly for themselves will continue, but most wont. They won't even become aware that such option exists.

Monkey see monkey do where people start these activities because they see others doing them will disappear entirely.

The move has happened because SEO rotted out The Internet to the point there’s been a “theory” that the Internet died a decade ago. If the current Internet is unsustainable under new technology trends then the new parts need to and will evolve to thrive inside the new ecosystem.

> It will be just a food for AI of some corporation.

Food that said corporation makes a profit off while paying the author nothing.

Especially with main SO I really wonder if accepting natural cycle of online platforms would be best. When culture gets to point that there is enough negative views on platform, maybe it is time to let it be replaced.

> In the short term it will feel liberating

They won't feel so liberated when they find ads embedded in the model's response in ways that make it difficult to uBlock.

> And that's assuming a world where people only ask LLMs and don't care about the provenance or trustworthiness of information at all. Which seems unlikely, even conspiracy nuts have some sources they trust more than others.

They will just have one model that they trust more because (a) it aligns with their views, or (b) it's a sycophant and agrees with anything and everything.

They definitely are not the people most likely to care about clicking "source links".

> People will have less or no motivation to create them

Not sure if we surf the same internets... In the web I am surfing, the more "motivation" (trying to get ad revenue) the author has, the crappier the content is. If I want to find high quality information, invariably I am seeking authors with no "motivation" whatsoever to produce the content (wikipedia, hacker news, reddit with a heavy filter etc.) I'm pretty sure we would be better off if the whole ad industry vanished.

Those are not easily searchable either

I hate that about Discord. It is a fun comms application, but it is severely lacking as a community.

Once, before I realized this, I recommended users of a forum use Discord. The impact was severe and fortunately brief. We all realized we would not be leaving the usual, often high value info for others, and ourselves to benefit from in the future.

We unwound that mess and now carry on in the usual way.

Discord has carved out a huge chunk of discussion people will wish was available in the future.

It depends very much on the discord. I'm in several discords which are excellent communities. They're really good places to hang out, get to know people interested in the same hobby, build a subculture etc.

I haven't really tried one as a QA or knowledge sharing site, perhaps they're much less good at that.

That explanation totally fails to account for SO's usage suddenly falling off a cliff as soon as ChatGPT arrived on the scene.

Yes, this is key. Business models based on generating low quality information and using SEO to monetise it will die and that's a good thing.

But unfortunately also sites that generate high quality information (eg independent research, reviews, journalism) will also struggle and be more reliant on paywalls and subscriptions.

Oh I know, it's one of my top 3 complaints. Information gardening is hard

Same. And my logs say that AI crawls my pages roughly 1.2 bazillion times per day.

Option (C) users railroaded into using a given model by a company or government with the power to make it a default or requirement.

Incoming uAIBlock. uBlockAi? The taglines write themselves.

> Stack Overflow for example is in a steep decline, only a small fraction of questions are posted today compared to the peak.

...which might be beneficial. A problem they'd been trying to deal with for over a decade was the massive influx of low quality duplicate and "do my homework for me" questions from people who don't even bother looking for a solution. If they've all moved off to AI things, problem solved and maybe SO can return to its high-quality origins?

They too busy editing questions to fit their answers.

at this point, i've built a massive platform for ten years and watching the seoers die might be worth it losing it all for. it'll at least kill off all the indians that have been scamming in my niche for three and half years with impunity

>Stack Overflow for example is in a steep decline

This is because they're a big bunch of assholes and no one wants to deal with that. Their decline started way before ChatGPT came in.

I fear we will get more walled gardens like FB because AI bots will have harder time scraping them.

Bottom line, you will always get content, whatever that may mean.

But great content creators will want to be rewarded one way or another. Book writers get paid, movie makers as well, so why shouldn't those who share their wisdom in a blog post? If someone is making money off your content and you aren't, you will not be happy. When it's not even attributed to you, it's even worse.

SEO is a response to Google's incentives, and Google can fix it if it wants.

because the answer to so many questions can be distilled from the content already on SO

Hopefully we have local AI by then, big tech will no doubt put filters in to stop you using their service to filter their own ads.

Frankly it’s not really great even for that, but they have captured the audience and now we are stuck with it.

Every time I boot into Manjaro to do some gaming, almost always there’s a new update for Discordavailable and guess what? The updates to Manjaro are always lagging behind a few days to a week, and Discord won’t run with a slightly out of date client.

The only way to get is working is using the snap, and who doesn’t want use some 3rd party package manager just to send some kbit/s voice data?

Additionally the interface sucks and is really bloated

With one quite significant issue IMO

If there's an old question the most upvoted answer will be at the top. Better solutions are often available if the previous answer was 10 years ago, but they will be buried.

Solution is obviously to scroll down as well as read the comments, but that can be time consuming.

I am with you on this the amount of answers I got from people that actually have knowledge about the libraries I am working with was always overshadowed by some semantic questioning. Do really need to? This tool might be better etc.

That's why I still prefer IRC without retention there is more freedom for discussion.

I suppose wanting to kill the commercial web is a valid position, although it feels more like grumpy old man yells at kids to get off his lawn than a considered analysis of relative value and impact, but even then I think you're underestimating the impact the AI problem will have on the non-commercial web as well.

Lots of people might be willing to run websites for fun or personal satisfaction or whatever, but how many people will continue to be willing to do so when they don't actually get to present the content to visitors and it's instead just regurgitated by AI? Half the fun of hosting your own website is personalizing it and choosing how to share the content. Even people blogging for fun put a lot of thought into their posts on how to phrase an argument or tell a story. But what's the point when nobody will ever see your actual post, just your thoughts rearranged and presented by AI? Maybe some people only care about the information being out there in any form, but I'd be willing to bet that's yet a smaller subset of even the people who would contribute in a return to geocities version of the web.

They can be excellent, but only in the moment.

The excellent communities I have been a part of can be searched. People can read it and learn. The Discord ones, unless they publish to a wiki or something, just don't exist.

Interesting point. The main one I'm part of started out as a wiki and is now both. The wiki is kind of the long term memory of the community, but it's so painful to have conversations there.