←back to thread

Understanding the BM25 full text search algorithm

(emschwartz.me)
283 points rrampage | 1 comments | 20 Nov 24 03:43 UTC | HN request time: 0.306s | source
Show context
hubraumhugo ◴[20 Nov 24 11:58 UTC] No.42193073[source]▶
Given the recent advances in vector-based semantic search, what's the SOTA search stack that people are using for hybrid keyword + semantic search these days?
replies(7): >>42193208 #>>42193787 #>>42193816 #>>42193909 #>>42193922 #>>42193932 #>>42194089 #
emschwartz ◴[20 Nov 24 12:15 UTC] No.42193208[source]▶
Most of the commercial and open source offerings for hybrid search seem to be using BM25 + vector similarity search based on embeddings. The results are combined using Reciprocal Rank Fusion (RRF).

The RRF paper is impressive in how incredibly simple it is (the paper is only 2 pages): https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf

replies(2): >>42193625 #>>42195796 #
softwaredoug ◴[20 Nov 24 16:51 UTC] No.42195796[source]▶
A warning that RRF is often not Enough, as it can just drag a good solution down towards the worse solution :)

https://softwaredoug.com/blog/2024/11/03/rrf-is-not-enough

replies(1): >>42196484 #
1. emschwartz ◴[20 Nov 24 17:54 UTC] No.42196484[source]▶
Ah, that's great! Thanks for sharing that.

I had actually implemented full text search + vector search using RRF but I kept it disabled by default because it wasn't meaningfully improving my results. This seems like a good hypothesis as to why.