Production RAG: what I learned from processing 5M+ documents

They say the chunker is the most important part, but theirs looks rudimentary: https://github.com/agentset-ai/agentset/blob/main/packages/e...

That is, there is nothing here that one could not easily write without a library.

OP here. We've been working on agentset.ai full-time for 2 months. The product now gets you something working quite well out of the box. Better than most people with no experience in RAG (I'd say so with confidence).

Ingestion + Agentic Search are two areas that we're focused on in the short term.

I'm not sure there is a chunker in this repo. The file you linked certainly doesn't seem to perform any chunking, it just defines a data model for chunks.

The only place I see that actually operates on chunks does so by fetching them from Redis, and AFAICT nothing in the repo actually writes to Redis, so I assume the chunker is elsewhere.

https://github.com/agentset-ai/agentset/blob/main/packages/j...