(emschwartz.me)

283 points rrampage | 1 comments | 20 Nov 24 03:43 UTC | HN request time: 0s | source

Show context

MPSimmons ◴[20 Nov 24 16:14 UTC] No.42195353[source]▶

>>42190650 (OP) #

Does anyone know if the average document length mentioned in the document length normalization is median? It seems like it would need to be to properly deweight excessively long documents, otherwise the excessively long documents would unfairly weight the average, right?

replies(1): >>42195523 #

1. softwaredoug ◴[20 Nov 24 16:28 UTC] No.42195523[source]▶

>>42195353 #

It’s the mean. At least in Lucene. Using median would be an interesting experiment.

Do you know of a search dataset with very large document length differences? MSMarco for example is pretty consistent in length.

↑

Understanding the BM25 full text search algorithm