←back to thread

Context is the bottleneck for coding agents now

(runnercode.com)
196 points zmccormick7 | 3 comments | 26 Sep 25 15:06 UTC | HN request time: 0.009s | source
Show context
aliljet ◴[26 Sep 25 15:27 UTC] No.45387614[source]▶
There's a misunderstanding here broadly. Context could be infinite, but the real bottleneck is understanding intent late in a multi-step operation. A human can effectively discard or disregard prior information as the narrow window of focus moves to a new task, LLMs seem incredibly bad at this.

Having more context, but leaving open an inability to effectively focus on the latest task is the real problem.

replies(10): >>45387639 #>>45387672 #>>45387700 #>>45387992 #>>45388228 #>>45388271 #>>45388664 #>>45388965 #>>45389266 #>>45404093 #
bgirard ◴[26 Sep 25 15:34 UTC] No.45387700[source]▶
I think that's the real issue. If the LLM spends a lot of context investigating a bad solution and you redirect it, I notice it has trouble ignoring maybe 10K tokens of bad exploration context against my 10 line of 'No, don't do X, explore Y' instead.
replies(6): >>45387838 #>>45387902 #>>45388477 #>>45390299 #>>45390619 #>>45394242 #
dingnuts ◴[26 Sep 25 15:47 UTC] No.45387838[source]▶
that's because a next token predictor can't "forget" context. That's just not how it works.

You load the thing up with relevant context and pray that it guides the generation path to the part of the model that represents the information you want and pray that the path of tokens through the model outputs what you want

That's why they have a tendency to go ahead and do things you tell them not to do..

also IDK about you but I hate how much praying has become part of the state of the art here. I didn't get into this career to be a fucking tech priest for the machine god. I will never like these models until they are predictable, which means I will never like them.

replies(8): >>45387906 #>>45387974 #>>45387999 #>>45388198 #>>45388215 #>>45388542 #>>45388863 #>>45390695 #
davedx ◴[26 Sep 25 15:59 UTC] No.45387999[source]▶
Yeah I start a new session to mitigate this. Don’t keep hammering away - close the current chat/session whatever and restate the problem carefully in a new one.
replies(2): >>45388047 #>>45388661 #
1. sethhochberg ◴[26 Sep 25 16:59 UTC] No.45388661[source]▶
I've had great luck with asking the current session to "summarize our goals, conversation, and other relevant details like git commits to this point in a compact but technically precise way that lets a new LLM pick up where we're leaving off".

The new session throws away whatever behind-the-scenes context was causing problems, but the prepared prompt gets the new session up and running more quickly especially if picking up in the middle of a piece of work that's already in progress.

replies(1): >>45388986 #
2. DenisM ◴[26 Sep 25 17:34 UTC] No.45388986[source]▶
Wow, I had useless results asking “please summarize important points of the discussion” from ChatGPT. It just doesn’t understand what’s important, and instead of highlighting pivoting moments of the conversation it produce a high level introduction for a non-practitioner.

Can you share you prompt?

replies(1): >>45389913 #
3. sethhochberg ◴[26 Sep 25 19:06 UTC] No.45389913[source]▶
Honestly, I just type out something by hand that is roughly like what I quoted above - I'm not big on keeping prompt libraries.

I think the important part is to give it (in my case, these days "it" is gpt-5-codex) a target persona, just like giving it a specific problem instead of asking it to be clever or creative. I've never asked it for a summary of a long conversation without the context of why I want the summary and who the intended audience is, but I have to imagine that helps it frame its output.