←back to thread

Understanding the BM25 full text search algorithm

(emschwartz.me)
283 points rrampage | 1 comments | 20 Nov 24 03:43 UTC | HN request time: 0s | source
Show context
RA_Fisher ◴[20 Nov 24 10:47 UTC] No.42192651[source]▶
BM25 is an ancient algo developed in the 1970s. It’s basically a crappy statistical model and statisticians can do far better today. Search is strictly dominated by learning (that yes, can use search as an input). Not many folks realize that yet, and / or are incentivized to keep the old tech going as long as possible, but market pressures will change that.
replies(4): >>42192735 #>>42192805 #>>42192828 #>>42194229 #
1. softwaredoug ◴[20 Nov 24 14:35 UTC] No.42194229[source]▶
I think there are also incentives to "sell new things". That's always been the case in search which has had a bazillion trends and "AI related things" as long as I've worked in it. We have massively VC funded vector search companies with armies of tech evangelists pushing a specific point of view right now.

Meanwhile, the amount of manual curation, basic, boring hand-curated taxonomies that actually drive things like "semantic search" at places like Google are simply staggering. Just nobody talks about them much at conferences because they're not very sexy.