←back to thread

El Capitan: New supercomputer is the fastest

(spectrum.ieee.org)
93 points rbanffy | 1 comments | 19 Nov 24 20:52 UTC | HN request time: 0s | source
Show context
pama ◴[19 Nov 24 21:37 UTC] No.42188372[source]▶
Noting here that 2700 quadrillion operations per second is less than the estimated sustained throughput of productive bfloat16 compute during the training of the large llama3 models, which IIRC was about 45% of 16,000 quadrillion operations per second, ie 16k H100 in parallel at about 0.45 MFU. The compute power of national labs has fallen far behind industry in recent years.
replies(3): >>42188382 #>>42188389 #>>42188415 #
alephnerd ◴[19 Nov 24 21:39 UTC] No.42188389[source]▶
Training an LLM (basically Transformers) is different workflow from Nuclear Simulations (basically Monte Carlo simulations)

There are a lot of intricates, but at a high level they require different compute approaches.

replies(3): >>42188413 #>>42188417 #>>42188497 #
handfuloflight ◴[19 Nov 24 21:42 UTC] No.42188413[source]▶
Can you expand on why the operations per second is not an apt comparison?
replies(1): >>42188538 #
1. pertymcpert ◴[19 Nov 24 21:57 UTC] No.42188538[source]▶
When you're doing scientific simulations, you're generally a lot more sensitive to FP precision than ML training which is very, very tolerant of reduced precision. So while FP8 might be fine for transformer networks, it would likely be unacceptably inaccurate/unusable for simulations.