Edit: I never actually expected AGI from LLMs. That was snark. I just think it's notable that the fundamental gains in LLM performance seem to have dried up.
And most would have accept the recommendation because the model sold it as less common tactic, while sounding very logical.
So many people just want to believe, instead of the reality of LLMs being quite unreliable.
Personally it's usually fairly obvious to me when LLMs are bullshitting probably because I have lots of experience detecting it in humans.
In this case I just happened to be domain expert and knew it was wrong. It would have required significant effort to verify everything with some less experienced person.