←back to thread

Ask HN: Is anyone doing anything cool with tiny language models?

684 points prettyblocks | 2 comments | 21 Jan 25 19:39 UTC | HN request time: 0.443s | source

I mean anything in the 0.5B-3B range that's available on Ollama (for example). Have you built any cool tooling that uses these models as part of your work flow?
Show context
psyklic ◴[21 Jan 25 20:05 UTC] No.42784612[source]▶
JetBrains' local single-line autocomplete model is 0.1B (w/ 1536-token context, ~170 lines of code): https://blog.jetbrains.com/blog/2024/04/04/full-line-code-co...

For context, GPT-2-small is 0.124B params (w/ 1024-token context).

replies(4): >>42785009 #>>42785728 #>>42785838 #>>42786326 #
pseudosavant ◴[21 Jan 25 22:22 UTC] No.42785838[source]▶
I wonder how big that model is in RAM/disk. I use LLMs for FFMPEG all the time, and I was thinking about training a model on just the FFMPEG CLI arguments. If it was small enough, it could be a package for FFMPEG. e.g. `ffmpeg llm "Convert this MP4 into the latest royalty-free codecs in an MKV."`
replies(4): >>42785929 #>>42786381 #>>42786629 #>>42787136 #
1. jedbrooke ◴[21 Jan 25 22:33 UTC] No.42785929[source]▶
the jetbrains models are about 70MB zipped on disk (one model per language)
replies(1): >>42794671 #
2. pseudosavant ◴[22 Jan 25 16:42 UTC] No.42794671[source]▶
That is easily small enough to host as a static SPA web app. I was first thinking it would be cool to make a static web app that would run the model locally. You'd make a query and it'd give the FFMPEG commands.