←back to thread

Heretic: Automatic censorship removal for language models

(github.com)
745 points melded | 4 comments | 16 Nov 25 15:00 UTC | HN request time: 0.961s | source
Show context
embedding-shape ◴[16 Nov 25 16:14 UTC] No.45946159[source]▶
Optuna is a generally useful project, that I'm surprised isn't used in more places in the ecosystem. The ability to do what they're doing here, incrementally find the best hyperparameter to use can really make a large difference in how quickly you can move past having to fine-tune those values. Basically any time you aren't sure about the perfect value, throw Optuna on it with a quick script, and make it go for a broad search first, then narrow it down, and you can let the computer figure out the best values.

Nicely done to pair that with something as fun as censorship removal, currently in the process on running it on gpt-oss-120b, eager to see the results :) I'm glad that someone seems to be starting to take the whole "lobotimization" that happens with the other processes seriously.

replies(3): >>45946248 #>>45946393 #>>45946623 #
1. Qwuke ◴[16 Nov 25 16:41 UTC] No.45946393[source]▶
I've seen Optuna used with some of the prompt optimization frameworks lately, where it's a really great fit and has yielded much better results than the "hyperparameter" tuning I had attempted myself. I can't stop mentioning how awesome a piece of software it is.

Also, I'm eager to see how well gpt-oss-120b gets uncensored if it really was using the phi-5 approach, since that seems fundamentally difficult given the training.

replies(1): >>45946603 #
2. p-e-w ◴[16 Nov 25 17:08 UTC] No.45946603[source]▶
FWIW, I already used Heretic to decensor gpt-oss-20b [1], and it works just fine. Note that the number of refusals listed on the model card is actually an overestimate because refusal trigger words occur in the CoT, even though the model doesn't actually end up refusing in the end.

[1] https://huggingface.co/p-e-w/gpt-oss-20b-heretic

replies(1): >>45946733 #
3. NitpickLawyer ◴[16 Nov 25 17:23 UTC] No.45946733[source]▶
What's your intuition on other "directions"? Have you tried it on something other than "refusals"? Say "correctness" in math or something like that. I have some datasets prepared for DPO on "thinking" traces that are correct / incorrect, wondering if it'd be something that could work, or if it's out of scope (i.e. correctness is not a single direction, like refusal training)
replies(1): >>45946858 #
4. p-e-w ◴[16 Nov 25 17:39 UTC] No.45946858{3}[source]▶
The problem is that in order to do optimization, you need a classifier that can distinguish the two types of responses (like refusal/compliance). In case of refusals, that's relatively easy to do using trigger words like "disallowed" or "I can't". I imagine this would be much, much harder to do automatically for classes like correctness.

And I also suspect, as you hint at, that "correctness" isn't just a direction in residual space, but a concept so broad that no simple mechanistic description can capture it.