OpenAI delays launch of open-weight model

To be completely and utterly fair, I trust Deepseek and Qwen (Alibaba) more than American AI companies.

American AI companies have shown they are money and compute eaters, and massively so at that. Billions later, and well, not much to show.

But Deepseek cost $5M to develop, and made multiple novel ways to train.

Oh, and their models and code are all FLOSS. The US companies are closed. Basically, the US ai companies are too busy treating each other as vultures.

Wasn’t that figure just the cost of the GPUs and nothing else?

Actually the majority of Google models are open source and they also were pretty fundamental in pushing a lot of the techniques in training forward - working in the AI space I’ve read quite a few of their research papers and I really appreciate what they’ve done to share their work and also release their models under licenses that allow you to use them for commercial purposes.

It was more than $5m

https://interestingengineering.com/culture/deepseeks-ai-trai...

$5 million was the gpu hour cost of a single training run.

Yeah, I hate that this figure keeps getting thrown around. IIRC, it's the price of 2048 H800s for 2 months at $2/hour/GPU. If you consider months to be 30 days, that's around $5.7M, which lines up. What doesn't line up is ignoring the costs of facilities, salaries, non-cloud hardware, etc. which will dominate costs, I'd expect. $100M seems like a fairer estimate, TBH. The original paper had more than a dozen authors, and DeepSeek had about 150 researchers working on R1, which supports the notion that personnel costs would likely dominate.

That is also just the final production run. How many experimental runs were performed before starting the final batch? It could be some ratio like 10 hours of research to every one hour of final training.

"Actually the majority of Google models are open source"

That's not accurate. The Gemini family of models are all proprietary.

Google's Gemma models (which are some of the best available local models) are open weights but not technically OSI-compatible open source - they come with usage restrictions: https://ai.google.dev/gemma/terms

Deepseek R1 was trained at least partially on the output of other LLMs. So, it might have been much more expensive if they needed to do it themselves from scratch.

> Billions later, and well, not much to show.

This is obviously false, I'm curious why you included it.

> Oh, and their models and code are all FLOSS.

No?

Lawsuit, since it was against OpenAI TOS: https://hls.harvard.edu/today/deepseek-chatgpt-and-the-globa...

You’re ignoring the T5 series of models that were incredibly influential, the T5 models and their derivatives (FLAN-T5, Long-T5, ByT5, etc) have been downloaded millions of times on huggingface and are real workhorses. There are even variants still being produced within the last year or so.

A yea the Gemma series is incredible and while maybe not meeting the standards of OSI - I consider them to be pretty open as far as local models go. And it’s not just the standard Gemma variants, Google is releasing other incredible Gemma models that I don’t think people have really even caught wind of yet like MedGemma, of which the 4b variant has vision capability.

I really enjoy their contributions to the open source AI community and think it’s pretty substantial.

Exactly. Not to minimize Deepseeks tremendous achievement, but that $5 million was just for the training run, not the GPUs used they purchased before, and all the OpenAI API calls they likely used to assist in synthetic data generation.

>ignoring the costs of facilities, salaries, non-cloud hardware, etc.

If you lease, those costs are amortized. It was definitely more than $5M, but I don't think it was as high as $100M. All things considered, I still believe Deepseek was trained at one (perhaps two) orders of magnitude lower cost than other competing models.

> But Deepseek cost $5M to develop, and made multiple novel ways to train

This is highly contested, and was either a big misunderstanding by everyone reporting it, or maliciously placed there (by a quant company, right before the stock fell a lot for nvda and the rest) depending on who you ask.

If we're being generous and assume no malicious intent (big if), anyone who has trained a big model can tell you that the cost of 1 run is useless in the big scheme of things. There is a lot of cost in getting there, in the failed runs, in the subsequent runs, and so on. The fact that R2 isn't there after ~6 months should say a lot. Sometimes you get a great training run, but no-one is looking at the failed ones and adding up that cost...

They were pretty explicit that this was only the cost in GPU hours to USD for the final run. Journalists and Twitter tech bros just saw an easy headline there. It's the same with Clair Obscur developer's Sandfall, where the people say that the game was made by 30 people, when there were 200 people involved.

Deepseek is far more worthy of the name OpenAI than Sam Altman's ClosedAI.

These "200 people" were counted from credits which list pretty much everyone who even sniffed at the general direction of the studio's direction. The studio itself is ~30 people (just went and check on their website, they have a team list with photos for everyone). The rest are contractors whose contributions usually vary wildly. Besides, credits are free so unless the the company are petty (see Rockstar not crediting people on their games if they leave before the game is released even if they worked on it for years) people err on the site on crediting everyone. Personally i've been credited on a game that used a library i wrote once and i learned about it years after the release.

Most importantly those who mention that the game was made by 30 people do it to compare it with other much larger teams with hundreds if not thousands of people and those teams use contractors too!

> They were pretty explicit that this was only the cost in GPU hours to USD for the final run.

The researchers? Yes.

What followed afterwards, I'm not so sure. There was clearly some "cheap headlines" in the media, but there were also some weird coverage being pushed everywhere, from weird tlds, and they were all pushing nvda dead, cheap deepseek, you can run it on raspberries, etc. That might have been a campaign designed to help short the stocks.

> American AI companies have shown they are money and compute eaters

Don't forget they also quite literally eat books

Perhaps. Do you think DeepSeek made use of those competing models at all in order to train theirs?

Who is literally eating books?