Ask HN: Is anyone doing anything cool with tiny language models?

I simply use it to de-anonymize code that I typed in via Claude

Maybe should write a plugin for it (open source):

1. Put in all your work related questions in the plugin, an LLM will make it as an abstract question for you to preview and send it

2. And then get the answer with all the data back

E.g. df[“cookie_company_name”] becomes df[“a”] and back

Could you recommend a tiny language model I could try out locally?

Llama 3.2 has about 3.2b parameters. I have to admit, I use bigger ones like phi-4 (14.7b) and Llama 3.3 (70.6b) but I think Llama 3.2 could do de-anonimization and anonimization of code

+1 this idea. I do the same. Just do it locally using ollama, also using 3.2 3b

Llama 3.2 punches way above its weight. For general "language manipulation" tasks it's good enough - and it can be used on a CPU with acceptable speed.

So you are using a local small model to remove identifying information and make the question generic, which is then sent to a larger model? Is that understanding correct?

I think this would have some additional benefits of not confusing the larger model with facts it doesn't need to know about. My erasing information, you can allow its attention heads to focus on the pieces that matter.

Requires further study.

How many tokens/s?

Are you using the model to create a key-value pair to find/replace and then reverse to reanonymize, or are you using its outputs directly? If the latter, is it fast enough and reliable enough?

You're using it to anonymize your code, not de-anonymize someone's code. I was confused by your comment until I read the replies and realized that's what you meant to say.

> So you are using a local small model to remove identifying information and make the question generic, which is then sent to a larger model? Is that understanding correct?

Yep that's it

I read it the other way, their code contains eg fetch(url, pw:hunter123), and they're asking Claude anonymized questions like "implement handler for fetch(url, {pw:mycleartrxtpw})"

And then claude replies

fetch(url, {pw:mycleartrxtpw}).then(writething)

And then the local llm converts the placeholder mycleartrxtpw into hunter123 using its access to the real code

10-15t/s on 12400 with ddr5

> Put in all your work related questions in the plugin, an LLM will make it as an abstract question for you to preview and send it

So the LLM does both the anonymization into placeholders and then later the replacing of the placeholders too. Calling the latter step de-anonymization is confusing though, it's "de-anonymizing" yourself to yourself. And the overall purpose of the plugin is to anonymize OP to Claude, so to me at least that makes the whole thing clearer.

It's that yea

Flow would be:

1. Llama prompt: write a console log statement with my username and password: mettamage, superdupersecret

2. Claude prompt (edited by Llama): write a console log statement with my username and password: asdfhjk, sdjkfa

3. Claude replies: console.log('asdfhjk', 'sdjkfa')

4. Llama gets that input and replies to me: console.log('mettamage', 'superdupersecret')

I could've been a bit more clear, sorry about that.