4-bit quantised model, to be precise.
When does this guy sleep?
replies(5):
When does this guy sleep?
and https://huggingface.co/TheBloke/Falcon-180B-Chat-GGUF if you want to try
Minimal overhead or zero cost abstractions around deep learning libraries implemented in those languages gives some hope that people like ggerganov are not afraid of the 'don't roll your own deep learning library' dogma and now we can see the results as to why DL on the edge and local AI, is the future of efficiency in deep learning.
We'll see, but Python just can't compete on speed at all, henceforth Modular's Mojo compiler is another one that solves the problem properly with the almost 1:1 familiarity of Python.
Historically average RAM has grown far faster than linear, and there really hasn't been anything pressing manufacturers to push the envelope here in the past few years... until now.
It could be that LLM model sizes keep increasing such that we continue to require cloud consumption, but I suspect the sizes will not increase as quickly as hardware for inference.
Given how useful GPT-4 is already. Maybe one more iteration would unlock the vast majority of practical use cases.
I think people will be surprised that consumers ultimately end up benefitting far more from LLMs than the providers. There's not going to be much moat or differentiation to defend margins... more of a race to the bottom on pricing
The problem is CUDA, not Python.
LLMs are uniquely suited to local inference in projects like GGML because they are so RAM bandwidth heavy (and hence relatively compute lite), and relatively simple. Your kernel doesn't need to be hyper optimized by 35 Nvidia engineers in 3 stacks before its fast enough to start saturating the memory bus generating tokens.
And yet its still an issue... For instance, llama.cpp is having trouble getting prompt ingestion performance in a native implementation comparable cuBLAS, even though they theoretically have a performance advantage by using the quantization directly.
There is no language war. Use whatever tool is necessary to achieve effective results for accomplishing the mission.
It seems a Thunderbolt/USB4 external NVME enclosure can do about 2500-3000 MB/s which is about half of internal SSD. So not at all bad. It’ll just add an additional few tens of seconds while loading the model. Totally manageable.
Edit: in fact this is the proper route anyway since it allows you to work with huge model and intermediate FP16/FP32 files while quantizing. Internal storage, regardless of how much, will run out quickly.
This only applies to Macs and Mac-a-likes. Actual desktop PCs have many SATA ports and can store reasonable amounts of data without the crutch of external high latency storage making things iffy. I say this as someone with TBs of llama models on disk and I do quantization myself (sometimes).
BTW my computer cost <$900 w/17TB of storage currently and can run up to 34B 5bit llm. I could spend $250 more to upgrade to 128GB of DDR4 2666 ram and run the 65B/70B but 180B is out of the range. You do have to spend big money for that.
Of course battery life would be a concern there, so I think LLM usage on phones will remain in the cloud.
Haven't studied phone RAM capacity growth rates in detail though
Or are you comparing with CPU inference? In which case apples-oranges.
How much do GPUs with 192GB of RAM cost?
Edit: also I think (unverified) very very few systems have multiple PCI 3/4 NVME slots. There are companies with PCI cards that can take NVMEs but that’ll in itself cost, without NVMEs, more than your $900 system.
Should be pointed out that this didn't just happen out of thin air. These open models still cost millions of dollars to create. Meta let the genie out of the bottle, but it won't be free forever.
Unless I'm misunderstanding, doesn't OpenAI have a very vested interest to keep making their products so good/so complex/so large that consumer hobbyists can't just `git clone` an alternative that's 95% as good running locally?
This particular model was funded by the UAE government. If they could do it, it should be similarly possible for a western government to create and release one as a public good.
The problem is the bus, cuda, and the sheer volume of data that need to be transferred.
Pytorch itself is actually a wrapper around torchlib, which is written in C++.
The compilation step of PyTorch 2.0 provides a sizeable improvement, but not 2 orders of magnitude as you’d expect from python to c++ migrations. The compilation is due to the backend more so than python itself. See Triton for example.
Wouldn’t be surprised if we get small models that can run locally on a phone and just retrieve data from the network as needed (without sending your data out), within the next couple of years
But I'm also in the quad-3090 build idea stage as well and bought 2 with the intention to go up to 4 eventually for this purpose. However, since I bought my first 2 a few months back (at about 800 euro each!) the ebay prices have actually gone up... a lot; I purchased a specific model that I thought would be plentiful as I had found a seller with a lot of them from OEM pulls, and they were taking good offers- and suddenly they all got sold. I feel like we are entering another GPU gap like 2020-2021.
Based on the performance of Llama2 70B, I think 96GB of vram and the cuda core count x bandwidth of 4 3090's will hit a golden zone as far as price-performance of a deep learning rig that can do a bit of finetuning on top of just inference.
Unless A6000 prices (or A100 prices) start plummeting.
My only hold out is the thought that maybe nvidia releases a 48gb Titan-type card at a less-than-A6000 price sometime soon, which will shake things up.
What I think will happen is that more companies will come to the realization it's in their best interest to open their giant models. The cost of training all those giant models is already a sunk cost. If there's no profit to be made by keeping a model proprietary, why not open it to gain or avoid losing mind-share, and to mess with competitors' plans?
First, it was LLaMA, with up to 65B params, opened against Meta's wishes. Then, it was LLaMA 2, with up to 70B params, opened by Meta on purpose, to mess with Google's and Microsoft/OpenAI's plans. Now, it's Falcon 180B. Like you, I'm wondering, what comes next?
From what I understand, if you take the absolute best cutting edge LLM with the most parameters and the most up to date model from GitHub/HuggingFace/whatever, it's very far off from the output you get from GPT-3.5 / GPT-4
aka full of hallucinations, not very useful
I don't know if this is the right way to look at it but if what George Hotz said about GPT-4 simply being "8 220B parameter models glued together by something called a mixture-of-experts", from what I understand, OpenAI's moat is:
their access/subsidiized cost to GPUs/infrastructure with Microsoft
the 8 220B models they have are really good/I don't think anything open source matches them/nobody can download "all of Reddit/Twitter/Wikipedia/StackOverflow/whatever else they trained on" anymore like they could given how everybody wants to protect/monetize their content now
and then the "router" / "MoE" piece seems to be something missing from open source offerings as well
Although see last week for previous responses: https://news.ycombinator.com/item?id=37333747
LLMs make possible the great skill sharing, they are learning from some people through web and books, and then assist other people in their particular problems. This level of sharing and customisation is even greater and more accessible than open source.
You generally can't hook up large storage drives to nvme. Those are all tiny flash storage. I'm not sure why you brought it up.
1) Need 2 power supplies to run at peak power and most US outlets can't handle it
2) If you're running on 1 power supply need to limit power consumption and clock speed of the cards to prevent the PSU fuse from popping
3) To get all 4 cards running at once there is an adjustment needed for most MB bios
4) Finding an MB that can handle enough PCI-e lanes can be a challenge
Ultimately I think I get 95% or higher performance on one PSU with appropriate power limiting and 48 lanes (vs a full 64)
I went with the cheapest threadripper I could find but after testing the performance hit for going from 16-8 PCIe lanes was actually not that large and I would be okay going with 8 lanes for each card.
What’s your definition of large?
2TB and 4TB NVME are not tiny. You can even buy 8TB NVMEs, though those are more expensive and IMHO not worth it for this use case.
2TB NVMEs are $60-$100 right now.
You can attach several of those via Thunderbolt/USB4 enclosures providing 2500-3000 MB/s
The user experience of working with language is terrible because most tasks it is utilized in go way beyond "scripting" scenario, which Python was primarily made for (aside from also being easy to pick up and use language).
I think you guys are missing a massive technical consideration which is cost. Training cost, offering cost. As with everything else in tech, outside of the bubble created by ZIRP over the last decade and a half (and the entire two generations of tech workers who never learned this important lesson thus far in their careers), costs matter and are a primary driver of technology success.
If you attached dollar costs to these models above, if the data was available, you’d quickly discover who (if anyone) has a sustainable business model and who doesn’t.
A sustainable model is what determines long term whether w technology is available and whether that leads to further improvement (and increasing sustainability/financial value).
Here’s another way to think about it. Why does ISA matter in CPUs? There are minor issues around efficiencies of various kinds, but the real advantage of any mainstream ISA is, in part, the availability of tooling (hence this was a correct and heavy early focus for the RISCV effort) but also a lot of ecosystem things you don’t see: for example, Intel and Arm have truly mammoth test and verification suites that represent thousands++ of man years of investment.
OpenAI almost certainly has a massive invisible accumulated value at this point.
The actual models themselves are the output in the same way that a packaged CPU is the output. How you got there matters almost as much or more.
1) I live in the EU with 240v mains and can pull 3600W from a single circuit and be OK. I'd also likely just limit the cards a bit for power efficiency as well as I do that already with a single 3090, running at 300W vs 350W max TDP makes such a small difference in performance I don't notice; this isn't gaming.
2) Still may run a dual PSU, but with the CPU below and the cards at 1200W i'd max at around 1600W, add overhead, and there are some 2000W PSUs that could do it. Also being on 240v mains makes this easier on the PSU selection as I have seen many can source more wattage when on 230/240 vs 115/120.
3) I'm leaning towards a Supermicro H12SSL-i motherboard which I've seen people run quad A5000's on without any hassle. Proper PCIe x16 slots spaces 2x apart, and i'd likely be watercooling all 4 cards together in a 2-slot spacing or installing MSI turbo heatsinks with blower fans / server rack style for 2 slot cooling.
4) See 3. AMD Epyc 7713 is currently this choice with 3200mhz ddr4 giving the memory bandwidth I need for a specific CPU workload as well.
I'm just currently trying to figure out if it is worth it dealing with the massive import duty I'd get hit with if I bought the H12SSL-i, RAM, and the EPYC from one of the many Chinese Ebay resellers that sell these things at basically wholesale. I get hit with 20% VAT plus another something like 20% duty on imported computer parts. It's close to the point where it would be cheaper for me to book a round trip flight to Taiwan and take parts home in my luggage. People in the USA don't realize how little they are paying for computing equipment, relatively.
So, minus the potential tax implications this can all be done for about 8000 EUR.
I do computational fluid dynamics work and am working on GPU-assisted solvers, which is the reason for the 64-core EPYC as parts of that workflow are still going to need CPU threads to pump the numbers to the GPGPUs. Also this is why I don't just run stuff in cloud like so many people will suggest when talking about spending 8K on a rig vs $2/hr per A100 at Lambda labs. My specific work needs to stay on-prem.
All told it is less than two A6000's at current prices and is a whole machine to boot.
One suggestion, also limit clock speeds/voltages. There are transients when the 3090 loads models that can exceed double their typical draw, 4 starting at once can draw substantially more than I expected.
Is that not why OpenAI is ahead right now? For free, you can have access to powerful AI on anything with a web browser. You don't need to wait for your SSD to load the model, page it into memory and swap your preexisting processes like it would on a local machine. You don't need to worry about the local battery drain, heat, memory constraints or hardware limitations. If you can read Hacker News, you can use AI.
Given the current performance of local models, I bet OpenAI is feeling pretty comfortable from where they're standing. Most people don't have mobile devices with enough RAM to load a 13b, 4-bit Llama quantization. Running a 180B model (much less a GPT-4 scale model) on consumer hardware is financially infeasible. Running it at-scale, in the cloud is pennies on the dollar.
I'm not fond of OpenAI in the slightest, but if you've followed the state of local models recently it's clear why they keep coming out ahead.
This is orders of magnitude lower than many companies and government R&D budgets. It's easily financeable by 1000s of independently wealthy people and organizations. It's easily financeable by VC money. This is far cheaper than many other startups or product initiatives that have been tried. There are very likely to be many organizations that build models for the specific purpose of open sourcing the resulting model... the Falcon and Llama models are already proof enough of this
Costs to train equivalent models may increase in the short term due to race towards GPU consumption raising costs... but compute will get cheaper in aggregate over time due to improving compute tech.
And once the model is built it is largely a sunk cost, yes. All that needs to happen is for a single SoTA model to be made open to completely negate any advantage a competitor has. Monetization from LLMs will be driven by focused application of the models, not from providing an interface to a general model. High quality data holds more value than the resulting model
Not every query requires timeliness of data. Incorporating new data into an existing model is likely to be cheaper than retraining the model from scratch, but just speculation on my end.
Privacy is the second case, I don't want to leak all my great ideas or data to openai or anyone else.
Meta is already capable of monetizing content generated by the models: these models complement their business and they could not care less which model you're using to earn them advertising dollars, as long as you keep the (preferably high quality) content coming.
As more and more models become open and are able to be run locally, the precedent gets stronger (which is good for the end consumer in my opinion).
Given that censorship is one of the biggest complaints about the hosted LLMs, it should be no surprise that some of the main use-cases driving local LLMs are those involving creating content that censored LLMs are unwilling to create.
What are some of key aspects about scenarios where this commodification happens? Where it doesn't?
Speaking descriptively (not normatively), I see a lot of possibilities about how things will unfold hinging on (a) licensing, (b) desire for recent data, (c) desire for private data, (d) regulation.
startup as legal entity has close to 0 value, most value is in intellectual property which is stored and transmitted by meatbags.
I had a 3x3090 rig. 2 of them are fried. 2 of them had two sided Bykski blocks (would never fit next to each others 2 slot apart, water header block with 90 degree corner plumbing is full 4x slot size minimum). 3090 are notorious for their vram chips, on the backside, failing due to insufficient cooling.
Also, 1kW of heat needs to be evacuated. Few hours of full power can easily heat up neighbouring rooms as well.
They sure didn't try very hard to secure it. I wonder if it was their strategy all along.
Then the inevitable occurred and made it obvious that the restrictions were both impractical to enforce and counterproductive, so they released a new one with less of them.
At which point the new underdogs would have an interest in doing to them what they're doing to OpenAI.
Assuming progress for LLMs continues at a rapid pace for an extended period of time. It's not implausible that they'll get to a certain level past which non-trivial progress is hard, and if there is an open source model at that level there isn't going to be a moat.
Honestly the answer is that it mostly doesn't.
An ISA isn't viable without tooling, but that's why it's the first thing they all get. The only ISA with any significant moat is x86, and that's because there is so much legacy closed source software for it that people still need but would have to be emulated on any other architecture. And even that only works as long as x86 processors are competitive; if they fell behind then customers would just eat the emulation overhead on something else.
Other ISAs don't even have that. Would anybody actually be surprised if RISC-V took a huge chunk of out ARM's market share in the not too distant future?
This is called a solid state drive.
And you could do the same thing without even changing the socket by including RAM on the CPU package as an L4 cache. Some of the Intel server CPUs are already doing this.
Other benchmarks/anecdotes suggest fine-tuned code models are outperforming GPT4 too. The trend seems to be that smaller, fine-tuned task specific models outperform larger generalised models. It requires a lot of resources to pretrain the base model, but as we’ve seen, there’s no shortage of companies who are willing and able to do that.
Not to mention, all those other companies are already profitable, whereas OpenAI is already burning investor cash.
M2 Ultra [0] seems to be max 295w
Dramatically oversimplifying of course. There will be niches where one will be the right choice over the other. In a continuous serving context you'd mostly only want to run models which can fully fit in the VRAM of a single 3090, otherwise the crosstalk penalty will apply. 24GB VRAM is enough to run CodeLlama 34B q3_k_m GGUF with 10000 tokens of context though.
The footage can be aggressively pruned to fit on the disk.
When the robot is delivered in 2033 it can easily figure out, from the footage, all these weird and rare edge cases.
The difference will be like that between a competent but new employee and someone with 10 years of experience.
I can see the Tesla bots disassembling the production line already. Or do you think it wont happen?
Obviously what you can do in practice is put the interface on your phone. It doesn't have to run on battery to run locally.
To do better than that would have required the version of Optane that used DIMM slots, which was something like a quarter of the performance of actual DRAM for half the price.
So you had something that costs more than ordinary SSDs if your priority is cost and is slower than DRAM if your priority is performance. A lot of times a middle ground like that is still valuable, but since cache hierarchies are a thing, having a bit of fast DRAM and a lot of cheap SSD serves that part of the market well too.
And in the meantime ordinary SSDs got faster and cheaper and DRAM got faster and cheaper. Now you can get older systems with previous generation DRAM that are faster than Optane for less money. They stopped making it because people stopped buying it.
(this is not financial advice and i am not a financial advisor.)
I have cut many hours of debugging thanks to it. I could find issues easily, on-call in short conversation, when previously that was reserved as post mortem task.
Even reading documentation is nothing like before: once, I was looking for a single command to upload and presign a object in S3. SDK has tens of methods, which require careful scanning, if they do what I want. Going through documentation thoroughly would've taken me hours. GPT-4 simply found, no, there's no operation for that immediately.