←back to thread

Production RAG: what I learned from processing 5M+ documents

(blog.abdellatif.io)
548 points tifa2up | 1 comments | 20 Oct 25 15:55 UTC | HN request time: 0.243s | source
Show context
esafak ◴[20 Oct 25 16:44 UTC] No.45645965[source]▶
They say the chunker is the most important part, but theirs looks rudimentary: https://github.com/agentset-ai/agentset/blob/main/packages/e...

That is, there is nothing here that one could not easily write without a library.

replies(2): >>45646130 #>>45646451 #
1. teraflop ◴[20 Oct 25 17:16 UTC] No.45646451[source]▶
I'm not sure there is a chunker in this repo. The file you linked certainly doesn't seem to perform any chunking, it just defines a data model for chunks.

The only place I see that actually operates on chunks does so by fetching them from Redis, and AFAICT nothing in the repo actually writes to Redis, so I assume the chunker is elsewhere.

https://github.com/agentset-ai/agentset/blob/main/packages/j...