No, it's not. The limitation is believing a human can define how the agent should recall things. Instead, build tools for the agent to store and retrieve context and then give it a tool to refine and use that recall in the way it sees best fits the objective.
Humans gatekeep, especially in the tech industry, and that is exactly what will limit us improving AI over time. It will only be when we turn over it's choices to it that we move beyond all this bullshit.