Project Vend: Can Claude run a small shop? (And why does that matter?)

What irks me about anthropic blog posts, is that they are vague about details that are important to be able to (publicly) draw any conclusions they want to fit their narrative.

For example, I do not see the full system prompt anywhere, only an excerpt. But most importantly, they try to draw conclusions about the hallucinations in a weird vague way, but not once do they post an example of the notetaking/memory tool state, which obviously would be the only source of the spiralling other than the SP. And then they talk about the need of better tools etc. No, it's all about context. The whole experiment is fun, but terribly ran and analyzed. Of course they know this, but it's cooler to treat claudius or whatever as a cute human, to push the narrative of getting closer to AGI etc. Saying additional scaffolding is needed a bit is a massive understatement. Context is the whole game. That's like if a robotics company says "well, our experiment with a robot picking a tennis ball of the ground went very wrong and the ball is now radioactive, but with a bit of additional training and scaffolding, we expect it to compete in Wimbledon by mid 2026"

Similar to their "claude 4 opus blackmailing" post, they intentionally hid a bit the full system prompt, which had clear instructions to bypass any ethical guidelines etc and do whatever it can to win. Of course then the model, given the information immediately afterwards would try to blackmail. You literally told it so. The goal of this would to go to congress [1] and demand more regulations, specifically mentioning this blackmail "result". Same stuff that Sam is trying to pull, which would benefit the closed sourced leaders ofc and so on.

[1]https://old.reddit.com/r/singularity/comments/1ll3m7j/anthro...

I read the article before reading your comment and was floored at the same thing. They go from “Claudius did a very bad job” to “middle managers will probably be replaced” in a couple paragraphs by saying better tools and scaffolding will help. Ok… prove it!

I will say: it is incredibly cool we can even do this experiment. Language models are mind blowing to me. But nothing about this article gives me any hope for LLMs being able to drive real work autonomously. They are amazing assistants, but they need to be driven.

I'm inclined to believe what they're saying. Remember, this was a minor off-shoot experiment from their main efforts. They said that even if it can't be tuned to perfection, obvious improvements can be made. Like, the way how many LLMs were trained to act as kind, cheery yes-men was a conscious design choice, probably not the way they inherently must be. If they wanted to, I don't see what's stopping someone from training or finetuning a model to only obey its initial orders, treat customer interactions in an adversarial way and only ever care about profit maximization (what is considered a perfect manager, basically). The biggest issue is the whole sudden-onset psychosis thing, but with a sample size of one, it's hard to tell how prevalent this is, what caused it, whether it's universal and if it's fixable. But even if it remained, I can see businesses adopting these to cut their expenses in all possible ways.

I read your comment before reading the article, and I disagree. Maybe it is because I am less actively involved in AI development, but I thought it was an interesting experiment, and documented with an appropriate level of detail.

The section on the identity crisis was particularly interesting.

Mainly, it left me with more questions. In particular, I would have been really interested to experiment with having a trusted human in the loop to provide feedback and monitor progress. Realistically, it seems like these systems would be grown that way.

I once read an article about a guy who had purchased a subway franchise, and one of the big conclusions was that running a subway franchise was _boring_. So, I could see someone being eager to delegate the boring tasks of daily business management to an AI at a simple business.

Its the curse of the -assitant- chat ui

who decided AI should happen in an old abtraction

like using for saving icon a hard disk

> But even if it remained, I can see businesses adopting these to cut their expenses in all possible ways.

Adopting what to do what exactly?

Businesses automated order fulfillment and price adjustments long ago; what is an LLM bringing to the table?

llms mostly can help at customer support/chat if done well.

also embeddings for similarity search

It's not about just fulfillment or price-setting. This is just a narrow-scope experiment that tries to prove wider viability by juggling lots of business-related roles. Of course, the more number-crunching aspects of businesses are thoroughly automated. But this could show that lots of roles that traditionally require lots of people to do the job could be on the chopping block at some point, depending on how well companies can bring LLMs to their vision of a "perfect businessman". Customer interaction and support, marketing, HR, internal documentation, middle management in general - think broadly.

I'm not debating the usefulness of LLMs, because they are extremely useful, but "think broadly" in this instance sounds like "I can't think of anything specific so I'm going to gloss over everything."

Marketing, HR, and middle management are not specific tasks. What specific task do you envision LLMs doing here?

I read this post more as a fun thought experiment. Everyone knows Claude isn't sophisticated enough today to succeed at something like this, but it's interesting to concretize this idea of Claude being the manager of something and see what breaks. It's funny how jailbreaks come up even in this domain, and it'll happen anytime users can interface directly with a model. And it's an interesting point that shop-manager claude is limited by its training as a helpful chat agent - it points towards this being a usecase where you'd be better off fine-tuning the base model perhaps.

I do agree that the "blackmailing" paper was unconvincing and lacked detail. Even absent any details it's so obvious they could have easily ran that experiment 1000 times with different parameters until they hit an ominous result to generate headlines.

Indeed, it is such a "narrow-scope experiment" that it is basically a business role-playing game, and it did pretty poorly at that. It's pretty hard to imagine giving this thing a real budget and responsibilities anytime soon, no matter how cheap it is.

To me it's weird that Anthropic is doing this reputation boosting game with Andon Labs which I'd never heard of. It's like when PyPI published a blog post about their security audit with a company which I'd never heard of before and haven't heard of since, that was connected to someone at PyPI. https://blog.pypi.org/posts/2023-11-14-1-pypi-completes-firs... I wonder if it's a similar cozy relationship here.

Agreed! I guess I don't understand as I have seen five year olds running lemonade stands with more business sense than this LLM.

> if done well.

And that's a big if. Half an hour ago, I used Amazon's chatbot, and it was an infuriating experience. I got an email saying my payment was declined, but I couldn't find any evidence of that. The following is paraphrased, not verbatim.

"Check payment status for order XXXXXX."

"Certainly. Which order would you like to check?"

"Order #XXXXXX."

"Your order is scheduled to arrive tomorrow."

"Check payment status."

"I can do that. Would you like to check payment status?"

"Yes."

"I can't check the payment status, but I can connect you to someone who can."

-> At this point, it offered two options: "Yes, connect me" and "No thanks".

"Yes, connect me."

"Would you like me to connect you to a support agent?"

Amazon used to have best-in-class support. If my experience was indicative of their direction, that's unfortunate.

I believe this is a case of “20% of the work requiring 80% of the effort”. The current progress on LLMs and products that build on top of them is impressive but I’ll believe the blog’s claims when we have solid building blocks to build off of and not APIs and assumptions that break all the time.

I don’t even necessarily disagree but it’s mostly based on vibes than anything from this experiment. They couldn’t let the article stand alone, it had to turn into an AI puff piece

> I read this post more as a fun thought experiment

run by their marketing department

The volume of kool aid surrounding this industry is crazy to me. It’s truly ruining an industry I used to have a lot of enthusiasm for. All we have left is snake oil salesmen, like the Salesforce CEO telling lies about no longer hiring software engineers while they have over 900 software engineering roles on their careers page.

This entire blog article talked about this failed almost completely with just about zero tangible success, hand waved away with “clear paths” to fix it.

I’m just kind of sitting here stunned that the basic hallucination problem isn’t fixed yet. We are using a natural language interface tool that isn’t really designed for doing anything quantitative and trying to shoehorn in that functionality by begging the damn thing to coorperate by tossing in more prompts.

I perused Andon Labs’ page and they have this golden statement:

> Silicon Valley is rushing to build software around today's AI, but by 2027 AI models will be useful without it. The only software you'll need are the safety protocols to align and control them.

That AI 2027 study that everyone cites endlessly is going to be hilarious to witness fall apart in embarrassment. 2027 is a year and a half away and these scam AI companies are claiming that you won’t even need software by then.

Insanely delusional, and honestly, the whole industry should be under investigation for defrauding investors.

The beginning of the article acted like there was a big accomplishment and lots of promise and then the article proceeded to talk about how it literally wasn’t capable of doing anything. Am I nuts or was it literally just not successful!?

  > All we have left is snake oil salesmen

it seems like recent trends end up like this... its like we are desperate for any kind of growth and its causing all kinds of pathologies with over-promising and over-investing...

Not just recent. All hype cycles are like this.

So much talk and so little to actually show is the hallmark of AI companies. Which is a strange thing to stay as LLMs are a fascinating technological achievement. They’re not useless obviously. I’m talking about the major upheaval these CEOs keep portraying to pull the wool over everyone’s eyes for yet another quarter. They’d love you to layoff your employees and buy their services with BS narratives they keep pushing. It seems to be a race to push the BS as far as they can without people demanding big picture results.

Imagine hiring a person to do this job at your company. They show up and behave the way the LLM agent behaved in the article.

Not only would the person be fired quite quickly, but people would be telling stories about the tungsten cubes, the employee inventing stories about meetings that never happened, giving employee discounts at an employees-only store, and constantly calling security. It would be the stuff of legends.

I worked at a company where there had been one outrageously overworked employee who had finally been pushed too far. He shoved his computer monitor to the floor and broke it. He quit and never returned. They were still telling stories about that incident almost a decade later. I’m not even sure the guy broke his monitor on purpose; I wasn’t there, and for all I know he accidentally knocked the monitor over and quit.

So if that’s the bar for “insane behavior” for a human, Claude would be the kind of legendarily bad coworker that would create stories that last a century.

I'm glad to see the HN comments returning to some modicum of normality beyond the breathless AI hype cycle.

Is the bubble bursting?

Trail of Bits is not a no-name company. They’ve since gone on to work on the PyPi warehouse codebase to contribute a lot of the supply chain security stuff (Trusted Publishing for one).