At one point, I gave it a scenario which demonstrated a common failure case, such an important one that it would have broken horribly in production. Its reaction was to make hundreds of changes, one of which, hidden behind hundreds of other changed lines, was to HARDCODE the special case which I had shown it.
Of course, that test then passed, and I assumed it had fixed the problem. It was only much later that I discovered this special-case handling. It was not caught during multiple rounds of AI code review.
Another instance of such a fuck-up was that the AI insisted on fixing tests which were failing, which it had written, but it kept continuously failing to do so. It ended up making hundreds of changes across various functions, sometimes related, sometimes unrelated, and never figured out that the test itself was not relevant and made no sense after a recent refactor. The AI completely failed to consider, after many rounds of back and forth and trying, to take a single step back and look at the function itself, instead of the line that was failing.
This happens every time I touch AIs and try to let them do work autonomously, regardless of which AI it is. People who think these AIs do a good job are the same people who would get chewed up during a 5 minute code review by a senior.
I am genuinely afraid for the horseshit quality ""work"" people who use AI extensively are outputting. I use AIs as a way to be more productive; if you use it to do your job for you, I pray for the people who have to use your software.