Why pay for a search engine

I hope Kagi succeeds. But personally, I think the web is dead. And no search engine can save it.

SEO spam has been strangling the web for years. Now, genAI SEO spam has escalated that onto inhuman levels. To make matters worse, no one wants to post to the open web anymore because their posts are just going to drown in that sea of spam and only genai's data stealing bots will read them. As the amount of spam posted to the web increases, the amount of worthwhile content posted decreases. Eventually, nothing of value will be posted. (like facebook?)

You can lay the blame for the web's death squarely at Google's feet for allowing SEO to hijack search in the first place (or maybe the government is to blame for not breaking up Google's ad/search empire fast enough). Either way, the big companies all know the end is here and are gambling on genai to replace search. Already, places of knowledge are closing their borders and charging fees for genai to access.

We have entered the internet's dark age.

Fun aside: I think it's hilarious and fitting that Google's genai model sucks. And I hope they lose the genai wars (just out of spite, not because I think any other genai is worth a shit).

> I hope Kagi succeeds. But personally, I think the web is dead. And no search engine can save it.

I concur. Maybe now is the time to seriously think about alternate visions for "search". I have been toying with an idea[1] along those lines myself out of sheer annoyance at the state of WWW and web "search" in general.

[1] https://nadh.in/blog/decentralised-open-indexes/

I do not blame Google for web’s current wasteland landscape, I blame ads and all the grifters who poison the landscape with SEO waste to try make some bucks with ads or other similar means.

Let say government nationalise google search or force to make it a non profit 15 years ago. How would that prevent SEO from happening? Isn’t SEO an inevitable outcome of website trying compete for attention? How would anyone prevent it from happening?

Google is an ad company that specifically lets SEO fill up its search results because those websites make them the most money per click. If Google continued to fight SEO abuse like they were literally founded to do then we wouldn’t be in this mess.

The same way people make adblockers happen. By making block lists of SEO garbage cruated by human beings.

Nextdns and the RPi alternative do it. Kagi has all the infrastructure in place to make it happen at the search level, it just requires more manual work right now.

You cite Facebook as an example of no meaningful content yet put all the blame upon Google? Maybe content creators are just incentivised into giving 90 percent of the public what they want.

The content added to Facebook was not what made the web what it was. The content indexed by Google, in turn, it was. The countless blogs or cooking recipes, fan sites, etc, were overriden with SEO filler slop over the years.

OP example of what a content-devoid internet would look like is exactly what Facebook is now. A site that started as the place to connect and share with your friends and family now is just a place that is filled with AI slop and emptiness.

It really is Google's fault. Those blogs SEO goals are only targeting top Google positions.

There's several examples where Google's SEO has actively made things worse, even before you look at ads. And to be clear, I do agree that advertising has also done massive damage to the web.

All the long, rambling stories you see before recipes? That's just there so the recipes can rank higher.

I can't remember the source, but I also remember a website adding AI generated slop to cater to Google's wishes, and leaving a page to explain to normal users why they did so.

It's also seen when you look at YouTube videos, where now it's basically necessary to have a clickbait thumbnail + title + reminders to like and subscribe.

> I think the web is dead

The web cannot 'die'. Everything and everyone moves online.

Point one: A return to pre-internet days is not likely.

Point two: Walled gardens had their peak moment. The dangers of everybody relying on them for our entire online existence are more than apparent to anybody with a firing neuron (and despite appearances, our species is not entirely inane).

Ergo. The only path is forward, a more sane, less monopolized web. Build it and they will come.

yes, yes, yes and yes!

> Eventually, nothing of value will be posted. (like facebook?)

I'm in a couple of dozen groups on Facebook and they produce interesting and thoughtful content every day.

Of course I wish they were on the open web, but don't blame the network.

The web is very much dead as a matter of fact. I don't think a search engine has he ability to filter out all the LLM garbage that's flooding the internet. Just look at stackoverflow and github. Ever since chatgpt was released, the contributions skyrocketed and the quality dropped off the face of the earth and into a dark oblivion.

I think that the core of the issue isn't "big corporations" but rather the fact that the internet is more accessible than ever, which means that the ratio of users who want to create content for fun, to the users who want to just consume content or create it for monetary reasons, is completely different than years ago.

what do you mean by "the web"? The web is all the people, companies, public sector etc. that are setting up web servers (whether self-hosted, cloud etc. it doesnt matter).

To say that the web is dead means that nobody outside massive social media type sites will still run a server (and SO / github are social media type walled gardens at this point, thats why they get enshittified).

But this development would be absurd as a whole. Mind you search engines as we have come to know them might be dead, but thats of their own making. People need to find / communicate useful, truthful information and sooner or later they will get it.

Well, you could consider a 'search engine' to be a searchable curated website directory. You could manually curate this directory and only use scrapers to allow text search on these websites. This will always be a possible option and a way ensure that your search site is good.

We are talking on a website on the web.

Not everything needs to go through a search engine.

>The web is very much dead as a matter of fact. I don't think a search engine has he ability to filter out all the LLM garbage that's flooding the internet.

this presupposes the argument that for the web to have value then you must have a search engine to allow you to find things of worth on the web.

Which argument you have not made but only taken as a given.

>Isn’t SEO an inevitable outcome of website trying compete for attention?

different Search engines with different rules and so forth would lessen the benefit of the O part, because it seems unlikely you could efficiently optimize for every particular algorithm and ruleset.

Perhaps SEO is an inevitable outcome of one big dominant search engine and also made worse by that search engine not really giving a shit about making its search work that well for years and years but only about how many ads they could push at people and how much they could charge for those ads.

That's an interesting idea.

While I don't have the answer, what we might consider is the incentives that come with each model.

In the current model, the incentives are clear. Google's incentive was to rent-seek all usefulness out of web search, privileging advertisement and their own profitability over usefulness.

I am not sure if a search engine beholden to the government would be ideal. Governments do have their own sets of interests (legitimate or otherwise) that may at times go against users.

Web search is in the end a piece of public infrastructure, used by billions across the world, very much subject to the Tragedy of the Commons.

Perhaps it should be some sort of nonprofit, as other projects are (Linux Foundation comes to mind, being a successful one).

> the web is dead [posted 3 hours ago on the web]

m-hm.

How do you square your idea that genai sucks with the fact that people are paying and using it, and not just for seo spam?

Pessimists live in a world built by optimists

Would you mind sharing links to those groups? (unless they are local to your community/region/etc.)

I figure if 90% of everything is crap, https://medium.com/luminasticity/90-crap-48e4c79419a9 then if the amount of available content has doubled in a decade, that is probably just too much crap to find the good stuff.

The open web as a concept doesn’t really make sense anymore, at least for written content.

Since by definition it will need to be open to the majority of the world’s population, regardless of writing skill, but the bottom half of writers are already indistinguishable from cutting edge LLM output.

And no one can install some sort of worldwide identity verification system, or it wouldn’t be ‘open’ anymore.

I agree with your assessment that maybe this is an inevitable outcome of one big dominant search engine.

But I would argue one big search engine is unavoidable since no one would want to use the second best search engine.

Google search didn’t come the best because the other service like mail map and YouTube. So breaking up Google does nothing to stop google search from being the monopoly it is today.

How would you define “SEO garbage”. Who has the power to decide what belongs in the list.

I would guess just the incentive for any website to become top of the search results would likely create the outcome of today’s SEO landscape.

If different search engines optimized for different niches, there would be an user base for "second best" search engines. For example, nothing beats Bing for porn search.

Except that Google is complicit and directly benefits of the current state of their search engine, no matter how awful it is to actually use it.

Are you talking about the quality of the prose or about the content of writing? An LLM can only act on information that has been put into it by people, while a real person can create new and important information – even a person with low literal or oratory skills.

After your post I tried to find dotnet groups in facebook, but all I found is spam and almost no humans, which again reminded me about dead internet theory. Could you please recommend some groups?

Both.

How do you differentiate between say 20 real people with low skills and LLM output lightly edited by a high skilled individual?

I think there may be a way forward for the internet. Why isn't hacker news being strangled by SEO spam for example? I think the way forward is a much smaller internet predicated on tight-knit communities approving everything that's shared. I don't know how this could scale but maybe that's the point, it must stay small to succeed.

If I could differentiate between it, then what the real person wrote or said would still have a separate value, even if the prose was bad.

Consider a normal criminal court case. Most witnesses don't know how to express themselves well neither in the written nor spoken word. Their testimony still holds value and gives important information.

But you can’t differentiate it… unless you have some method that you can share?

Of course we can separate fact from fiction if we need: By checking up things ourselves. There is only 1 reality and only 1 truth.

What power? The power to push a github commit? That’s up to each blocklist owner.

There is no one list to rule them all. Tons of people curate their own block lists and make them available. It’s entirely up to you to pick and choose which ones you want to use that most align with your views on SEO garbage. You can override them with your own preferences too, as I do all the time with NextDNS. Like I said, all the infrastructure to support this is already in place in Kagi, they just need to implement the support for external lists.

For example, here is a list of the crowdsourced blocklists available in uBlock Origin: https://github.com/gorhill/ublock/wiki/Filter-list-licenses

Why wouldn’t the fake output mimic real human output after some light editing?

And it’s not like a live video call, where a few seconds delay would be noticeable.

There would still be a real human being, just as smart as you, behind the LLM, but pretending to be say 20 different lower skilled people.

Yes it would mimic real human output. That's why we have to verify if the subject matter is important. Knowing who to trust or not is not easy, if we have to be sure we have to do some work.

I still don’t get how this helps you differentiate between them… or do you mean to assume both are genuine human outputs regardless, if the information proves to be genuinely true?

> Let say government nationalise google search or force to make it a non profit 15 years ago. How would that prevent SEO from happening? Isn’t SEO an inevitable outcome of website trying compete for attention? How would anyone prevent it from happening?

I've said this before (on HN) and I'll say it again: a search engine that refuses to index sites containing advertisements absolutely kills off SEO.

What's the point of getting to the top of the search results if you are unable to monetise it?

The only way forward is to refuse to index sites with advertisements. Google will obviously not do this (and, in fact, to me it looks like they do the reverse - downrank non-monetised content - because it's in their best interest to serve sites with ads).

If the information proves to be true, what does it matter?

> What's the point of getting to the top of the search results if you are unable to monetise it?

Promoting your product/service, see brand astroturfing in Reddit.

> Promoting your product/service, see brand astroturfing in Reddit.

But ... that's the point, is it not? Someone searching for "FOO" will find all those sites who optimised for selling FOO.

That's an improvement over finding the top 10 results all optimised on FOO but delivering ads to make money, not FOO.

Because your interlocutors would be 20 people that don’t exist… and would take up 20x more of your time than a single low skilled individual.

Yes, even scoundrels may supply to you true information for some period of time but eventually they will try to obtain their actual goals…