Project Vend: Can Claude run a small shop? (And why does that matter?)

What irks me about anthropic blog posts, is that they are vague about details that are important to be able to (publicly) draw any conclusions they want to fit their narrative.

For example, I do not see the full system prompt anywhere, only an excerpt. But most importantly, they try to draw conclusions about the hallucinations in a weird vague way, but not once do they post an example of the notetaking/memory tool state, which obviously would be the only source of the spiralling other than the SP. And then they talk about the need of better tools etc. No, it's all about context. The whole experiment is fun, but terribly ran and analyzed. Of course they know this, but it's cooler to treat claudius or whatever as a cute human, to push the narrative of getting closer to AGI etc. Saying additional scaffolding is needed a bit is a massive understatement. Context is the whole game. That's like if a robotics company says "well, our experiment with a robot picking a tennis ball of the ground went very wrong and the ball is now radioactive, but with a bit of additional training and scaffolding, we expect it to compete in Wimbledon by mid 2026"

Similar to their "claude 4 opus blackmailing" post, they intentionally hid a bit the full system prompt, which had clear instructions to bypass any ethical guidelines etc and do whatever it can to win. Of course then the model, given the information immediately afterwards would try to blackmail. You literally told it so. The goal of this would to go to congress [1] and demand more regulations, specifically mentioning this blackmail "result". Same stuff that Sam is trying to pull, which would benefit the closed sourced leaders ofc and so on.

[1]https://old.reddit.com/r/singularity/comments/1ll3m7j/anthro...

I read the article before reading your comment and was floored at the same thing. They go from “Claudius did a very bad job” to “middle managers will probably be replaced” in a couple paragraphs by saying better tools and scaffolding will help. Ok… prove it!

I will say: it is incredibly cool we can even do this experiment. Language models are mind blowing to me. But nothing about this article gives me any hope for LLMs being able to drive real work autonomously. They are amazing assistants, but they need to be driven.

I'm inclined to believe what they're saying. Remember, this was a minor off-shoot experiment from their main efforts. They said that even if it can't be tuned to perfection, obvious improvements can be made. Like, the way how many LLMs were trained to act as kind, cheery yes-men was a conscious design choice, probably not the way they inherently must be. If they wanted to, I don't see what's stopping someone from training or finetuning a model to only obey its initial orders, treat customer interactions in an adversarial way and only ever care about profit maximization (what is considered a perfect manager, basically). The biggest issue is the whole sudden-onset psychosis thing, but with a sample size of one, it's hard to tell how prevalent this is, what caused it, whether it's universal and if it's fixable. But even if it remained, I can see businesses adopting these to cut their expenses in all possible ways.