←back to thread

ETH Zurich and EPFL to release a LLM developed on public infrastructure

(ethz.ch)
505 points andy99 | 1 comments | 11 Jul 25 18:45 UTC | HN request time: 0.216s | source
Show context
isusmelj ◴[11 Jul 25 20:30 UTC] No.44536509[source]▶
I hope they do well. AFAIK they’re training or finetuning an older LLaMA model, so performance might lag behind SOTA. But what really matters is that ETH and EPFL get hands-on experience training at scale. From what I’ve heard, the new AI cluster still has teething problems. A lot of people underestimate how tough it is to train models at this scale, especially on your own infra.

Disclaimer: I’m Swiss and studied at ETH. We’ve got the brainpower, but not much large-scale training experience yet. And IMHO, a lot of the “magic” in LLMs is infrastructure-driven.

replies(4): >>44536696 #>>44536809 #>>44537201 #>>44539869 #
1. andy99 ◴[11 Jul 25 21:09 UTC] No.44536809[source]▶
Imo, a lot of the magic is also dataset driven, specifically the SFT and other fine tuning / RLHF data they have. That's what has separated the models people actually use from the also-rans.

I agree with everything you say about getting the experience, the infrastructure is very important and is probably the most critical part of a sovereign LLM supply chain. I would hope there will also be enough focus on the data, early on, that the model will be useful.