←back to thread

291 points meetpateltech | 1 comments | | HN request time: 0s | source
Show context
lysecret ◴[] No.45957078[source]
Im pretty deep into this topic and what might be interesting to an outsider is that the leading models like neuralgcm/weathernext 1 before as well as this model now are all trained with a "crps" objective which I haven't seen at all outside of ml weather prediction.

Essentially you add random noise to the inputs and train by minimizing the regular loss (like l1) and at the same time maximizing the difference between 2 members with different random noise initialisations. I wonder if this will be applied to more traditional genai at some point.

replies(5): >>45957193 #>>45958032 #>>45958654 #>>45960250 #>>45962842 #
nerdponx ◴[] No.45958654[source]
> Essentially you add random noise to the inputs and train by minimizing the regular loss (like l1) and at the same time maximizing the difference between 2 members with different random noise initialisations. I wonder if this will be applied to more traditional genai at some point.

We recently had a situation where we specifically wanted to generate 2 "different" outputs from an optimization task and struggled to come up with a good heuristic for doing so. Not at all a GenAI task, but this technique probably would have helped us.

replies(1): >>45962869 #
1. albertzeyer ◴[] No.45962869[source]
This idea is often used for self-supervised learning (SSL). E.g. see DINO (https://arxiv.org/abs/2104.14294).