Will Amazon S3 Vectors kill vector databases or save them?

"I recently spoke with the CTO of a popular AI note-taking app who told me something surprising: they spend twice as much on vector search as they do on OpenAI API calls. Think about that for a second. Running the retrieval layer costs them more than paying for the LLM itself. That flips the usual assumption on its head." Hmm well start sending full documents as part of context see it flip back :).

Egress costs? I’m really surprised by this. Thanks for sharing.

Sry maybe should've being more clear it was a sarcastic remark. The whole point of doing vector db search is to feed LLM with very targeted context so you can save $ on API calls to LLM.

That’s not the whole point it’s in the intersection of reducing tokens sent but also getting search both specific and generic enough to capture the correct context data.

It's possible to create linking documents between the documents to help smooth out things in some cases.

They’re likely using an HNSW index, which typically requires a lot of memory for large data sets.

if they use AzureSearch, I fully understand it. Those things are hella expensive

No worries. I should probably make sure I have at least a token understanding of the topic cloud based architecture before commenting next time haha.