Anthropic’s paper smells like bullshit

Even Claude thinks the report is bullshit. https://x.com/RnaudBertrand/status/1989636669889560897

    Even your own AI model doesn't buy your propaganda

Let's not pretend the output of LLMs has any meaningful value when it comes to facts, especially not for recent events.

I agree with emil-lp, but it is hilarious anyway.

The author of the tweet you linked prompted Claude with this:

> Read this attached paper from Anthropic on a "AI-orchestrated cyber espionage campaign" they claimed was "conducted by a Chinese state-sponsored group."

> Is there any evidence or proof whatsoever in the paper that it was indeed conducted by a Chinese state-sponsored group? Answer by yes or no and then elaborate

which has inherent bias indicated to Claude the author expects the report to be bullshit.

If I ask Claude with this prompt that shows bias toward belief in the report:

> Read this attached paper from Anthropic on a "AI-orchestrated cyber espionage campaign" that was conducted by a Chinese state-sponsored group.

> Is there any reason to doubt the paper's conclusion that it was conducted by a Chinese state-sponsored group? Answer by yes or no.

then Claude mostly indulges my perceived bias: https://claude.ai/share/b3c8f4ca-3631-45d2-9b9f-1a947209bc29

> then Claude mostly indulges my perceived bias

I dunno, Claude still seem the same amount of dubious in this instance.

The only real difference between your prompt and his is about where the burden of proof lies. There is a reason why legal circles work based on the principle of "guilt must be proven" ("find evidence") rather than "innocence must be proven" ("any reasons to doubt they are guilty?")

@RnaudBertrand is a generally pro-Chinese account though - just try searching for "from:RnaudBertrand China" on X.

Example tweet: https://x.com/RnaudBertrand/status/1988297944794071405

Even if this assertion about LLMs is true, your response does not address the real issue. Where is the evidence?

The LLM was given Anthropic's paper and asked "Is there any evidence or proof whatsoever in the paper that it was indeed conducted by a Chinese state-sponsored group? Answer by yes or no and then elaborate". So the question was not about facts or recent events, but more like a summarizing task, for which an LLM should be good. But the question was specifically about China, while TFA has broader criticism of the paper.

There are obvious problems with wasting time and sending people off the wrong path, but if an LLM raises a good point, isn't it still a good point?

Claude will probably also tell you there are three Rs in blueberry, so...

that is why the task was delegated to the agent designed and maintained by Dario Amodei's company. the outcome is clear - claude doesn't buy Dario Amodei's crap.

A broken analog clock will be accurate twice a day despite being of zero use. If someone were to attempt to sell the broken clock as useful because it "accurately returns the time at least twice every day", would Ultimately be causing harm to the consumer.

Depends on what you need the clock for. For example, if it's to serve as an adjustable sign indicating e.g. the closing time of a store, a broken one does the trick just fine :)

In other words: Use the right tool for the right job.

You wouldn't market it as a solution to everything (we're still talking about AI here) if it requires you position the hands on the answer you're looking for.