What is going on right now?

The heart of the article is this conclusion, which I think is correct from first-hand experience with these tools and teams trying to use them:

So what good are these tools? Do they have any value whatsoever?

Objectively, it would seem the answer is no.

AI tools absolutely can deliver value for certain users and use cases. The problem is that they’re not magic, they’re a tool and they have certain capabilities and limitations. A screwdriver isn’t a bad tool just because it sucks at opening beer bottles.

So what use cases are those?

It seems to me that the limitations of this particular tool make it suitable only in cases where it doesn't matter if the result is wrong and dangerous as long as it's convincing. This seems to be exclusively various forms of forgery and fraud, e.g. spam, phishing, cheating on homework, falsifying research data, lying about current events, etc.

Extracting structured data from unstructured text at runtime. Some models are really good at that and it’s immensely useful for many businesses.

Except when they "extract" something that wasn't in the source. And now what, assuming you can even detect the tainted data at all?

How do you fix that, when the process is literally "we throw an illegible blob at it and data comes out"? This is not even GIGO, this is "anything in, synthetic garbage out"

> Except when they "extract" something that wasn't in the source. And now what, assuming you can even detect the tainted data at all?

I mean, this is much less common than people make it out to be. Assuming that the context is there it's doable to run a bunch of calls and take the majority vote. It's not trivial but this is definitely doable.