←back to thread

291 points meetpateltech | 1 comments | | HN request time: 0s | source
Show context
lysecret ◴[] No.45957078[source]
Im pretty deep into this topic and what might be interesting to an outsider is that the leading models like neuralgcm/weathernext 1 before as well as this model now are all trained with a "crps" objective which I haven't seen at all outside of ml weather prediction.

Essentially you add random noise to the inputs and train by minimizing the regular loss (like l1) and at the same time maximizing the difference between 2 members with different random noise initialisations. I wonder if this will be applied to more traditional genai at some point.

replies(5): >>45957193 #>>45958032 #>>45958654 #>>45960250 #>>45962842 #
albertzeyer ◴[] No.45962842[source]
The random noise is added to the model parameters, not the inputs, or not?

This reminds me of variational noise (https://www.cs.toronto.edu/~graves/nips_2011.pdf).

If it is random noise on the input, it would be like many of the SSL methods, e.g. DINO (https://arxiv.org/abs/2104.14294), right?

replies(1): >>45965559 #
1. lysecret ◴[] No.45965559[source]
Yes you are right it's applied to the parameters, but other models (like ngcm) applied it to the inputs. IMO it shouldn't make a huge difference main point is you max differences between models.