←back to thread

Olmo 3: Charting a path through the model flow to lead open-source AI

(allenai.org)
361 points mseri | 4 comments | 21 Nov 25 06:50 UTC | HN request time: 0.2s | source
Show context
Oras ◴[21 Nov 25 17:57 UTC] No.46006894[source]▶
I got excited by reading the article about releasing the training data, went to their HF account to look at the data (dolma3) and first rows? Text scraped from porn websites!

https://huggingface.co/datasets/allenai/dolma3

replies(2): >>46007075 #>>46008470 #
1. andy99 ◴[21 Nov 25 20:15 UTC] No.46008470[source]▶
Isn’t this before any curation has happened? I looked at it, I can see why it looks bad, but if they’re really being open about the whole pipeline, they have to include everything. Giving them a hard time for it only promotes keeping models closed.

That said I like to think of it was my dataset I would have shuffled that part down in the list so it didn’t show up on the hf preview

replies(1): >>46008534 #
2. Oras ◴[21 Nov 25 20:20 UTC] No.46008534[source]▶
Hard time? What value does adult videos description, views and comments add to small (7,32B) models?
replies(2): >>46008582 #>>46009574 #
3. andy99 ◴[21 Nov 25 20:26 UTC] No.46008582[source]▶
It says it’s common crawl, I interpret it to mean this is a generic web scrape dataset, presumably they filter stuff out they don’t want before pretraining. You’d have to do do some ablation testing to know what value it adds
4. khimaros ◴[21 Nov 25 22:09 UTC] No.46009574[source]▶
what if that's where they learned how to utilize the double entendre? hard times indeed.