Apertus 70B: Truly Open - Swiss LLM by ETH, EPFL and CSCS

Report: https://github.com/swiss-ai/apertus-tech-report/raw/refs/hea...

Key features

Fully open model: open weights + open data + full training details including all data and training recipes

Massively Multilingual: 1811 natively supported languages

Compliant: Apertus is trained while respecting opt-out consent of data owners (even retrospectivey), and avoiding memorization of training data

Their struggle with Nvidia driver bugs they had to work around was very relatable. You'd think if someone buys 10,752 of their high-end GPUs you'd get some support with it.

did I miss a blog on this?

we didn't have time to write one yet, but there is the tech report which has a lot of details already

Report is packed with interesting details. Engineering challenges and solutions chapter especially show how things which are supposed and expected to work break when put through a massive scale. Really difficult bugs. Great writeup.