←back to thread

Context is the bottleneck for coding agents now

(runnercode.com)
196 points zmccormick7 | 3 comments | 26 Sep 25 15:06 UTC | HN request time: 0.006s | source
Show context
aliljet ◴[26 Sep 25 15:27 UTC] No.45387614[source]▶
There's a misunderstanding here broadly. Context could be infinite, but the real bottleneck is understanding intent late in a multi-step operation. A human can effectively discard or disregard prior information as the narrow window of focus moves to a new task, LLMs seem incredibly bad at this.

Having more context, but leaving open an inability to effectively focus on the latest task is the real problem.

replies(10): >>45387639 #>>45387672 #>>45387700 #>>45387992 #>>45388228 #>>45388271 #>>45388664 #>>45388965 #>>45389266 #>>45404093 #
bgirard ◴[26 Sep 25 15:34 UTC] No.45387700[source]▶
I think that's the real issue. If the LLM spends a lot of context investigating a bad solution and you redirect it, I notice it has trouble ignoring maybe 10K tokens of bad exploration context against my 10 line of 'No, don't do X, explore Y' instead.
replies(6): >>45387838 #>>45387902 #>>45388477 #>>45390299 #>>45390619 #>>45394242 #
dingnuts ◴[26 Sep 25 15:47 UTC] No.45387838[source]▶
that's because a next token predictor can't "forget" context. That's just not how it works.

You load the thing up with relevant context and pray that it guides the generation path to the part of the model that represents the information you want and pray that the path of tokens through the model outputs what you want

That's why they have a tendency to go ahead and do things you tell them not to do..

also IDK about you but I hate how much praying has become part of the state of the art here. I didn't get into this career to be a fucking tech priest for the machine god. I will never like these models until they are predictable, which means I will never like them.

replies(8): >>45387906 #>>45387974 #>>45387999 #>>45388198 #>>45388215 #>>45388542 #>>45388863 #>>45390695 #
1. dragonwriter ◴[26 Sep 25 15:57 UTC] No.45387974[source]▶
This is where the distinction between “an LLM” and “a user-facing system backed by an LLM” becomes important; the latter is often much more than a naive system for maintaining history and reprompting the LLM with added context from new user input, and could absolutely incorporate a step which (using the same LLM with different prompting or completely different tooling) edited the context before presenting it to the LLM to generate the response to the user. And such a system could, by that mechanism, “forget” selected context in the process.
replies(2): >>45388257 #>>45388827 #
2. yggdrasil_ai ◴[26 Sep 25 16:24 UTC] No.45388257[source]▶
I have been building Yggdrasil for that exact purpose - https://github.com/zayr0-9/Yggdrasil
3. PantaloonFlames ◴[26 Sep 25 17:18 UTC] No.45388827[source]▶
At least a few of the current coding agents have mechanisms that do what you describe.