←back to thread

Meta's open AI hardware vision

(engineering.fb.com)
212 points GavCo | 1 comments | 15 Oct 24 18:08 UTC | HN request time: 0.54s | source
Show context
TechDebtDevin ◴[15 Oct 24 19:43 UTC] No.41852366[source]▶
> "This effort pushed our infrastructure to operate across more than 16,000 NVIDIA H100 GPUs, making Llama 3.1 405B the first model in the Llama series to be trained at such a massive scale."

So at 20k a pop (assuming meta has a decent wholesale price from Nividia) they spent $320 MILLION on the 405B model (not including probably 5-10 million in electricity for the training process, water, staff, infra).

Do we think that brings more than 400+ million in value to Meta? I think so. I don't want to do the math, so I'll ask Perplexity to look it up:

> "How much has Meta's valuation increased since they released their first open source model"

Answer (edited):

> Closing price on February 23, 2023: $509.50 > Closing price on October 11, 2024: $573.68 > The increase in stock price is $64.18 per share. > Total increase = Price increase per share × Number of outstanding shares > Total increase = $64.18 × 2,534,000,000 = $162,632,100,000 > Meta's stock valuation has increased by approximately $162.63 billion since the release of their first open source model on February 24, 2023.

They seem to be making the right choices!

replies(10): >>41852405 #>>41852476 #>>41852557 #>>41852616 #>>41852649 #>>41853410 #>>41853466 #>>41853527 #>>41853550 #>>41853606 #
1. asdff ◴[15 Oct 24 22:00 UTC] No.41853550[source]▶
I wonder how this math works in light of Moore's law? E.g say it cost meta $320 million to train this model this year. How much does it cost to train that model next year or the year after instead? Is it significantly cheaper? Are the returns on investment the same? Makes me think there is a business case in watching someone spend a pile of money to train model X, waiting to see if there's interest in the market in this model X, then spend a comparatively smaller pile of money on the same model X yourself taking advantage of lower future costs of compute and undercut the original company hand over fist.