Improving LLM token usage when debugging

When debugging with an LLM, a failed build sends ~200 tokens of mostly useless output. The actual error? Maybe 60 tokens. Multiply that by 20-30 commands per debugging session, and you're burning through tokens like crazy.

So, I created a CLI tool that acts as a smart filter between your commands and the LLM. It knows what errors look like across different tech stacks and only shows what matters.

Before:

``` bash > npm run build:graphql && react-router typegen && tsc && react-router build

> build:graphql > graphql-codegen

Parse Configuration Generate outputs app/features/tasks/services/atoms.ts:55:60 - error TS2339: Property 'taskId' does not exist on type '{ request: UpdateTaskRequest; }'.

55 const response = await apiClient.updateTask(params.taskId, params.request); ~~~~~~

Found 1 error in app/features/tasks/services/atoms.ts:55 ```

After:

```bash $ aex frontend-build app/features/tasks/services/atoms.ts(55,60): error TS2339: Property 'taskId' does not exist Done ```

That's it. When the build succeeds? Just "Done" - literally 1 token instead of 200.

Have a look! The full article is here: https://github.com/byme8/apparatus.exec/discussions/1

Code page at github https://github.com/byme8/apparatus.exec

I like the idea of this and think it could be useful, but I think this version of it isn't the right one.

It looks like you get little to no parameterization of commands. [edit] There is a "description" field for commands, but the recommended way to show command usage is to run an invalid command?[/edit] As it stands, it's a half-baked command runner.

The other main feature is basically grep presets, but they are too limiting. It looks like you can extract lines, but cannot get preceding/following lines, cannot specify a start/end inclusion pattern, or anything like that. Trivial example: I want line starting with "[ERROR]" and all following lines that start with a space (this is a typical log-line-continuation format).

So, maybe when refining this, drop the command runner support entirely and integrate with an existing one. For example, "aex npm test -- tests/foo.test.ts" is a wrapper around that command, and aex now has enough information to select which grep preset to use (the "npm test" one). And then extend what I can actually do with that preset.

Good luck!

I think that the easy solution to this is to run the command with a cheap model in a sub agent that is going to detect the relevant information and raise it to the main context. I think claude code is already doing this.

Trying to find the perfect set of regex that will cover all the edge cases for all the programs+versions is quite a task.