←back to thread

Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

(github.com)
343 points sillysaurusx | 4 comments | 05 Mar 23 04:28 UTC | HN request time: 0s | source
Show context
linearalgebra45 ◴[05 Mar 23 11:25 UTC] No.35028638[source]▶
It's been enough time since this leaked, so my question is why aren't there blog posts already of people blowing their $300 of starter credit with ${cloud_provider} on a few hours' experimentation running inference on this 65B model?

Edit: I read the linked README.

> I was impatient and curious to try to run 65B on an 8xA100 cluster

Well?

replies(2): >>35028936 #>>35030027 #
v64 ◴[05 Mar 23 12:15 UTC] No.35028936[source]▶
The compute necessary to run 65B naively was only available on AWS (and perhaps Azure, I don't work with them) and the required instance types have been unavailable to the public recently (it seems everyone had the same idea to hop on this and try to run it). In my other post here [1], the memory requirements have been lowered through other work, and it should now be possible to run the 65B on a provider like CoreWeave.

[1] https://news.ycombinator.com/item?id=35028738

replies(2): >>35029106 #>>35029766 #
linearalgebra45 ◴[05 Mar 23 12:40 UTC] No.35029106[source]▶
Are you sure about that? I can't remember where I saw the table of memory requirements, but I'm sure some of the larger instances here [1] will surely be able to cope (assuming they're available!)

Oracle gives you a $300 free trial, which equates to running BM.GPU4.8 for over 10 hours - enough for a focused day of prompting

[1] https://www.oracle.com/cloud/compute/gpu/

replies(3): >>35029110 #>>35030261 #>>35034167 #
1. v64 ◴[05 Mar 23 12:41 UTC] No.35029110[source]▶
> Are you sure about that?

I'm not. The only way to know it is to try :) thank you for the link!

replies(1): >>35029159 #
2. linearalgebra45 ◴[05 Mar 23 12:49 UTC] No.35029159[source]▶
You only get a single month-long window to spend the credit! And I'm sure not going to spend any of my own money on prompting experiments.

I might be suffering from FOMO to some degree, I've just got to tell myself that this won't have been the only time model weights get leaked!

replies(1): >>35030979 #
3. mynameisvlad ◴[05 Mar 23 16:20 UTC] No.35030979[source]▶
> And I'm sure not going to spend any of my own money on prompting experiments.

This certainly sounds a lot like whining that others aren’t doing the work you yourself don’t want to do.

replies(1): >>35031273 #
4. linearalgebra45 ◴[05 Mar 23 16:47 UTC] No.35031273{3}[source]▶
"prompting experiments" is just my use-case. According to v64 a lot of people have had the same idea of spinning up a trial instance to run inference, which is unsurprising.

I'm not in a position to put in any meaningful work towards optimising this model for lower-end hardware, or working on the tooling/documentation/user experience.