←back to thread

623 points magicalhippo | 2 comments | | HN request time: 0.54s | source
Show context
derbaum ◴[] No.42620643[source]
I'm a bit surprised by the amount of comments comparing the cost to (often cheap) cloud solutions. Nvidia's value proposition is completely different in my opinion. Say I have a startup in the EU that handles personal data or some company secrets and wants to use an LLM to analyse it (like using RAG). Having that data never leave your basement sure can be worth more than $3000 if performance is not a bottleneck.
replies(6): >>42621036 #>>42621592 #>>42622470 #>>42622485 #>>42622500 #>>42622740 #
lolinder ◴[] No.42622470[source]
Heck, I'm willing to pay $3000 for one of these to get a good model that runs my requests locally. It's probably just my stupid ape brain trying to do finance, but I'm infinitely more likely to run dumb experiments with LLMs on hardware I own than I am while paying per token (to the point where I currently spend way more time with small local llamas than with Claude), and even though I don't do anything sensitive I'm still leery of shipping all my data to one of these companies.

This isn't competing with cloud, it's competing with Mac Minis and beefy GPUs. And $3000 is a very attractive price point in that market.

replies(2): >>42623584 #>>42624306 #
1. logankeenan ◴[] No.42624306[source]
Have you been to the localLlama subreddit? It’s a great resource for running models locally. It’s what got me started.

https://www.reddit.com/r/LocalLLaMA/

replies(1): >>42624609 #
2. lolinder ◴[] No.42624609[source]
Yep! I don't spend much time there because I got pretty comfortable with llama before that subreddit really got started, but it's definitely turned up some helpful answers about parameter tuning from time to time!