Show HN: Semantic grep with local embeddings

(github.com)

177 points Runonthespot | 1 comments | 07 Sep 25 11:20 UTC | HN request time: 0s | source

Show context

CuriouslyC ◴[07 Sep 25 18:55 UTC] No.45161101[source]▶

I actually have a WIP library for this, the indexing server isn't where I want it just yet, but I have an entire agent toolkit that does this stuff, and the indexing server is quite advance, with self-tuning, raptor/lsp integration, solves for optimal result set using knapsack, etc.

https://github.com/sibyllinesoft/grimoire

replies(1): >>45162275 #

threecheese ◴[07 Sep 25 21:21 UTC] No.45162275[source]▶

>>45161101 #

I have to know, what is the Lens SPI? The link in your readme is broken, and Kagi results for this cannot possibly be right.

replies(1): >>45162926 #

1. CuriouslyC ◴[07 Sep 25 22:51 UTC] No.45162926[source]▶

>>45162275 #

Lens is basically a rust local first mmapped file base search store, it combines RAPTOR with LSP, semantic vectors and a dual dense/sparse encoding, and can learn a function over those to tune the weights of the relevance sources adaptively per query using your data. It also uses linear programming to select an "efficient" set of results that minimizes mutual information between result atoms -- regular rag/rerank pipelines just dump the top K, but those often have a significant amount of overlap so you bloat context for no benefit.

↑