92% reduction is amazing. I often write product marketing materials for devtool companies and load llms.txt into whatever AI I’m using to get accurate details and even example code snippets. But that instantly adds 60k+ tokens which, at least in Google AI Studio, annoyingly slows things down. I’ll be trying this.
Edit: After a longer look, this needs more polish. In addition to key question raised by someone else about quality, there are signs of rushed work here. For example the critical llm_min_guideline.md file, which tells the LLM how to interpret the compressed version, was lazily copy-pasted from an LLM response without even removing the LLM's commentary:
"You are absolutely right! My apologies. I was focused on refining the detail of each section and overlooked that key change in your pipeline: the Glossary (G) section is no longer part of the final file..."
Doesn't exactly instill confidence.
Really nice idea. I hope you keep going with this as it would be a very useful utility.
replies(1):