←back to thread

Ask HN: Is anyone doing anything cool with tiny language models?

684 points prettyblocks | 4 comments | 21 Jan 25 19:39 UTC | HN request time: 0.209s | source

I mean anything in the 0.5B-3B range that's available on Ollama (for example). Have you built any cool tooling that uses these models as part of your work flow?
Show context
antonok ◴[21 Jan 25 23:57 UTC] No.42786841[source]▶
I've been using Llama models to identify cookie notices on websites, for the purpose of adding filter rules to block them in EasyList Cookie. Otherwise, this is normally done by, essentially, manual volunteer reporting.

Most cookie notices turn out to be pretty similar, HTML/CSS-wise, and then you can grab their `innerText` and filter out false positives with a small LLM. I've found the 3B models have decent performance on this task, given enough prompt engineering. They do fall apart slightly around edge cases like less common languages or combined cookie notice + age restriction banners. 7B has a negligible false-positive rate without much extra cost. Either way these things are really fast and it's amazing to see reports streaming in during a crawl with no human effort required.

Code is at https://github.com/brave/cookiemonster. You can see the prompt at https://github.com/brave/cookiemonster/blob/main/src/text-cl....

replies(4): >>42786891 #>>42786896 #>>42793119 #>>42793157 #
binarysneaker ◴[22 Jan 25 00:03 UTC] No.42786891[source]▶
Maybe it could also send automated petitions to the EU to undo cookie consent legislation, and reverse some of the enshitification.
replies(3): >>42786953 #>>42787244 #>>42788894 #
1. K0balt ◴[22 Jan 25 00:43 UTC] No.42787244[source]▶
I think there is real potential here, for smart browsing. Have the llm get the page, replace all the ads with kittens, find non-paywall versions if possible and needed, spoof fingerprint data, detect and highlight AI generated drivel, etc. The site would have no way of knowing that it wasn’t touching eyeballs. We might be able to rake back a bit of the web this way.
replies(1): >>42787340 #
2. antonok ◴[22 Jan 25 00:58 UTC] No.42787340[source]▶
You probably wouldn't want to run this in real-time on every site as it'll significantly increase the load on your browser, but as long as it's possible to generate adblock filter rules, the fixes can scale to a pretty large audience.
replies(2): >>42788192 #>>42794640 #
3. K0balt ◴[22 Jan 25 02:30 UTC] No.42788192[source]▶
I was thinking running it in my home lab server as a proxy, but yeah, scaling it to the browser would require some pretty strong hardware. Still, maybe in a couple of years it could be mainstream.
4. Tepix ◴[22 Jan 25 16:39 UTC] No.42794640[source]▶
Depends on your machine and on the LLM. Could be doable.