Personally, I have some notes and bookmarks that I'd like to scrape, then have an LLM summarize, generate hierarchical tags, and store in a database. For the notes part at least, I wouldn't want to give them to another provider; even for the bookmarks, I wouldn't be comfortable passing my reading profile to anyone.
This is one of the reason why recently I added floating chat to https://recurse.chat/ to quickly access local LLM.
Here's a demo: https://x.com/recursechat/status/1846309980091330815
That being said, I think the more straightforward approach would be to utilize an existing library like https://github.com/collabora/WhisperLive/ within a Docker container. This way, you can call it via WebSocket and integrate it with my LLM, which could also serve as a nice feature in your product.
I've actually been playing around with speech to text recently. Thank you for the pointer, docker is a bit too heavy to deploy for desktop app use case but it's good to know about the repo. Building binaries with Pyinstaller could be an option though.
Real time transcription seems a bit complicated as it involves VAD so a feasible path for me is to first ship simple transcription with whisper.cpp. large-v3-turbo looks fast enough :D
Depends on the model, but in general, no.
...but it's fine for simple 1 liner commands like "how do I revert my commit?" or "rename these files to camelcase".
> How often does it fail?
Immediately and constantly if you ask anything hard.
An 8b model is not chat-gpt. The 3B model in the OP post is not chat-gpt.
The capability compared to sonnet/4o is like a potato and a car.
Search for 'LLM Leaderboard' and you can see for yourself. The 8b models do not even rank. They're generally not capable enough to use as a self hosted assistant.
This is not true. On benchmarks, maybe, but I find the LLM Arena more accurately accounts for the subjective experience of using these things, and Llama 3.1 8B ranks relatively high, outperforming GPT-3.5 and certain iterations of 4.
Where the 8Bs do struggle is that they don't have as deep a repository of knowledge, so using them without some form of RAG won't get you as good results as using a plain larger model. But frankly I'm not convinced that RAG-free chat is the future anyway, and 8B models are extremely fast and cheap to run. Combined with good RAG they can do very well.
But yes, like the guy above said it's really only helpful for one line commands. Like if I forgot some sort flag thats available for a certain type of command. Or random things I don't work with often enough to memorize their little build commands etc. It's not helpful for programming just simple commands.
It also can help with unstructured or messy data to make it more readable, although there's potential to hallucinate if the context is at all large.
> 8B models are extremely fast and cheap to run
yes.
> Combined with good RAG they can do very well.
This is simply not true. They perform at a level which is useful for simple, trivial tasks.
If you consider that 'doing well', then sure.
However, if, like the parent post, you want to be writing scripts, which is specifically what they asked... then: heck, what 8B are you using, because llama 3.1 is shit at it out of the box.
¯\_(ツ)_/¯
A working unit test can take 6 or 7 iterations with a good prompt. Forget writing logic. Creating classes? Using RAG to execute functions from a spec? Forget it.
That's not not the level that I need for an assistant.