Meta's open AI hardware vision

> "This effort pushed our infrastructure to operate across more than 16,000 NVIDIA H100 GPUs, making Llama 3.1 405B the first model in the Llama series to be trained at such a massive scale."

So at 20k a pop (assuming meta has a decent wholesale price from Nividia) they spent $320 MILLION on the 405B model (not including probably 5-10 million in electricity for the training process, water, staff, infra).

Do we think that brings more than 400+ million in value to Meta? I think so. I don't want to do the math, so I'll ask Perplexity to look it up:

> "How much has Meta's valuation increased since they released their first open source model"

Answer (edited):

> Closing price on February 23, 2023: $509.50 > Closing price on October 11, 2024: $573.68 > The increase in stock price is $64.18 per share. > Total increase = Price increase per share × Number of outstanding shares > Total increase = $64.18 × 2,534,000,000 = $162,632,100,000 > Meta's stock valuation has increased by approximately $162.63 billion since the release of their first open source model on February 24, 2023.

They seem to be making the right choices!

> Do we think that brings more than 400+ million in value to Meta?

Tough to tell, given nobody is turning a net profit on LLMs yet.

Companies have a tendency to develop neuroses, though, just like people. Apple’s near miss with bankruptcy fuelled cash hoarding. For Facebook, their disastrous IPO and near miss of mobile seems to have made them hyper aware of the Innovator’s Dilemma. $400mm spent on a defensive move is certainly wider than tens of billions on the metaverse.

Do you think that the 16k GPUs get used once and then are thrown away? Llama 405B was trained over 56 days on the 16k GPUs; if I round that up to 60 days and assume the current mainstream hourly rate of $2/H100/hour from the Neoclouds (which are obviously making margin), that comes out to a total cost of ~$47M. Obviously Meta is training a lot of models using their GPU equipment, and would expect it to be in service for at least 3 years, and their cost is obviously less than what the public pricing on clouds is.

> Tough to tell, given nobody is turning a net profit on LLMs yet.

I suspect in the case of Meta and other big players, profit isn't necessary required to bring substantial value. Imagine their model being able to help them moderate more fairly and accurately. This alone could prevent potential legal actions from individuals, companies, and governments.

> profit isn't necessary required to bring substantial value

They’re private companies. If they can’t tie it to profit, it’s not adding value.

> being able to help them moderate more fairly and accurately

This reduces legal costs and increases strategic flexibility. Sort of like HR or legal departments: cost centres add value by controlling costs, a critical component of profitability.

> Meta's stock valuation has increased by approximately $162.63 billion

All thanks to their investments in VR! Wait guys where are you going

One of the things driving this crazy spend on AI infrastructure by big tech is excess capital.

Big tech is getting increasing scrutiny by regulators, etc for monopolistic/anticompetitive practices/etc[0]. From S&P:

"Regulatory scrutiny is likely to translate into fewer mega deals, however. Adobe Inc. recently abandoned its $20 billion acquisition of Figma Inc. in the face of ongoing opposition from the UK's Competition and Markets Authority and the EU's European Commission. While Microsoft Corp. was able to close its $68.7 billion Activision Blizzard buy, it took nearly two years to complete the deal amid an intense fight with the regulators.

The harsh regulatory environment will continue to have a chilling effect on large tech M&A, but strategic buyers are likely to focus on smaller tuck-ins that do not tempt regulators to intervene."

So instead of growing further by spend on acquiring/rolling out new platforms to draw further scrutiny, they're shoveling piles of cash towards other capital expenses (with accounting tricks) that don't get further attention/scrutiny from regulators.

Meta doesn't use this infrastructure just for LLaMA - they get to use these hundreds of millions of dollars of spend to extract more value (deeper, not wider) from the users and platforms they already have.

Case in point: AI generated ads[1]. There's a not-too-distant future where the ads displayed on Facebook are an AI generated video of you in a new Toyota (or whatever). In terms of even open models Meta is much more than LLaMA. They have done a lot with speech (seamless), vision (segment anything), and much more.

That model training, inference, etc is running on these Nvidia GPUs as well.

There is also a take from Reid Hoffman saying that buying 100,000 GPUs or whatever is basically "table stakes" at this point[2]. Boards, analysts, the market, etc says "What I know for sure is this AI stuff needs Nvidia GPUs. Elon has 100k. How many do we have?".

So it's also a Cold War arms race/missile gap scenario. Plus, don't discount ego/"male anatomy measuring contest" that goes on with these guys. Consider this story where a bunch of centi-billionaires got together at Nobu to fight over Nvidia GPU shipments[3].

[0] - https://www.spglobal.com/marketintelligence/en/news-insights...

[1] - https://www.theverge.com/2024/10/8/24265065/meta-ai-edited-v...

[2] - https://www.theinformation.com/articles/openai-investor-hoff...

[3] - https://finance.yahoo.com/news/elon-musk-oracles-larry-ellis...

SPY has gone from 404 to 579 in that timeframe - so meta lost value and performance due to their choices? Or you're using a terrible metric to judge things by.

Providing toilets to employees does not tie to profit. Toilets in all offices are now closed. We are saving over $10k per day.

> They’re private companies. If they can’t tie it to profit, it’s not adding value.

Not true, even strictly from an accounting perspective. If you spend $400M and build an asset that is worth >$400M, then you have increased the value of the company without modifying profit. For example, if a company were to buy land, and build a building, that building has a value regardless of if it is associated with any revenue.

Some people are gonna shit in their pants hearing this new policy.

> Tough to tell, given nobody is turning a net profit on LLMs yet

Perhaps, but Meta is definitely getting some money back from ad impressions supported by AI-generated content.

Correct. We know these models are producing fucktonnes of revenue. At least some of them can be run at a gross profit, i.e. where marginal power costs and capital costs are less than marginal revenues. (Put another way: if OpenAI were an absolute monopoly and stopped training new models, could it turn a profit?)

What’s unclear is if this is a short-term revenue benefit from people fucking around with the newest, shiniest model, or recurring revenue that is only appearing unprofitable because the frontier is advancing so quickly.

> * If you spend $400M and build an asset that is worth >$400M, then you have increased the value of the company without modifying profit. For example, if a company were to buy land, and build a building, that building has a value regardless of if it is associated with any revenue*

Why does it have value? It’s because it can be rented, occupied for productive use or sold to someone who can do either of those. Revenue-free assets are essentially money. (Companies aren’t in the business of non-revenue non-monetary assets—that’s the domain of society at large.)

And Meta is using a lot of GPUs for offline ML and online ML features on Instagram, FB etc. So nothing is "wasted".

The stock price is completely wrong, stock was around $170 back in February 2023

> They’re private companies. If they can’t tie it to profit, it’s not adding value.

Smart companies understand other types of value exist.

If a democratic population hates you, it is harder to convince politicians to do your bidding. (Not impossible, just harder!)

If potential employees don't think kindly of you, it is harder to recruit.

Llama is a constant source of good PR for Meta in the developer community. Compared to just a couple of years ago when they were mostly laughed at by devs for metaverse stuff. Now it is "holy cow Zuck is standing up to Microsoft and Amazon and democratizing AI!"

With Llama, Meta has got great PR, and also developed cutting edge tech.

They also get to benefit from thousands of developers trying to make Meta's models run more efficiently.

From the little we know about OpenAIs inference infra, I feel like I can confidently say that if training stopped today, and they got cut off Azure subsidies, their $20.00 subscription model would probably not cover the cost of Inference.

I know nothing about the enterprise side of OpenAI but I'm sure they're profitable there. I doubt the subscription cost of a single power user of ChatGPT Plus covers the water they consume as a single user (This is probably an exaggeration, but I think I'm in the ballpark).

> I don't want to do the math, so I'll ask Perplexity to look it up

These numbers are totally wrong, and it takes about 30 seconds to look it up. It closed at 589.95 on 2024-10-11 and 172.04 on 2023-02-23. The other numbers appear to be wrong too.

It may be that extra-large LLMs don’t make sense for ChatGPT. They’re for enterprise use, like supercomputers. The reason I say “at least some” is I’ve found use in running a local instance of Llama, which seems to imply there is some niche of (legal) activities AI can support sustainably. (Versus, e.g. crypto.)

I did a double-take at the quoted $500 price in 2023. The real numbers strengthen the case for AI as a pump-the-stock investment.

Meta seems to - in my view correctly - understand that the risk to Facebook & other Meta properties is someone else walking away with great success, capturing the market for generative AI & being left out.

They don't need to make money, increase their value. They need to ward off existential risk. The best way to insure they aren't locked out from the future is to keep the future open. That's the only hope they have to maintain the locks they already have on much of the information technology world.

I wonder how this math works in light of Moore's law? E.g say it cost meta $320 million to train this model this year. How much does it cost to train that model next year or the year after instead? Is it significantly cheaper? Are the returns on investment the same? Makes me think there is a business case in watching someone spend a pile of money to train model X, waiting to see if there's interest in the market in this model X, then spend a comparatively smaller pile of money on the same model X yourself taking advantage of lower future costs of compute and undercut the original company hand over fist.

Please don't post auto-generated text without checking it yourself. It is almost always wrong, as it is here.

Another good reason not to use perplexity

> Providing toilets to employees does not tie to profit. Toilets in all offices are now closed. We are saving over $10k per day

Right. How do those companies tend to wind up?

I didn't say short-term profits. I said ultimately, the value of a non-monetary assets are tied to profitability. Particularly financial assets, e.g. C corporations. That doesn't mean that's the only measure of value. But for a company it's damn close.

Put another way: when a for-profit company starts arguing that profits don't matter, it's a little bit curious.

> other types of value exist

Sure. But they intermediate to profit. I'm not suggesting leadership should be justifying everything in those terms. But if an entire product line doesn't have a solid long-term net revenue generating or cost saving consequence, it's a flag for governance.

> potential employees don't think kindly of you, it is harder to recruit

And you get higher churn. Cost imperative. Not the sole reason--you have to work with the people, after all. But that's an agent benefit. Companies treat high-value employees well because it makes sense to, and they'd probably cease to exist if they stopped doing it.