‘Overworked, underpaid’ humans train Google’s AI

"Google said in a statement: “Quality raters are employed by our suppliers and are temporarily assigned to provide external feedback on our products. Their ratings are one of many aggregated data points that help us measure how well our systems are working, but do not directly impact our algorithms or models.” GlobalLogic declined to comment for this story." (emphasis mine)

How is this not a straight up lie? For this to be true they would have to throw away labeled training data.

They probably don’t do it at a scale large enough to do RLHF with it, but it’s still useful feedback the people working on the projects / products.

Because they are doing it to compute quality metrics not to implement RLHF. It’s not training data.

More recent models actually use "reinforcement learning from AI feedback", where the task of assigning a reward is essentially fed back into the model itself. Human feedback is then only used to ground the training, on selected examples (potentially even entirely artificial ones) where the AI is most highly uncertain about what feedback should be given.

Key word: "directly"

It does so indirectly, so it's a true albeit misleading statement.

> For this to be true they would have to throw away labeled training data.

That's how validation works.

Is there a reason not to use validation data in your next round of training data? Or is it more efficient to reuse validation and instead get more training data?

Every decision they take based on evals influences the model.

You'd have to recreate your validation if you trained your model on it every iteration and then they wouldn't be consistent enough to show any trends

It's not part of the inner feedback loop. It's part of the outer feedback loop that they use to decide if the inner loop is working.

/"directly"/

I'd have thought that if you kept the same validation you'd risk over fitting.

Clearly that does make it hard to measure. I'd think you'd want "equivalent" validation (like changing the SATs every year), though I imagine that's not really a meaningful concept.