←back to thread

Zamba2-7B

(www.zyphra.com)
282 points dataminer | 10 comments | 14 Oct 24 22:45 UTC | HN request time: 0s | source | bottom
1. jwitthuhn ◴[15 Oct 24 01:20 UTC] No.41843985[source]▶
For anyone else looking for the weights which as far as I can tell are not linked in the article:

Base model: https://huggingface.co/Zyphra/Zamba2-7B

Instruct tuned: https://huggingface.co/Zyphra/Zamba2-7B-Instruct

replies(1): >>41844057 #
2. keyle ◴[15 Oct 24 01:33 UTC] No.41844057[source]▶
I couldn't find any gguf files yet. Looking forward to trying it out when they're available.
replies(3): >>41844163 #>>41844635 #>>41847638 #
3. alchemist1e9 ◴[15 Oct 24 01:55 UTC] No.41844163[source]▶
What can be used to run it? I had imagined Mamba based models need a different interference code/software than the other models.
replies(3): >>41844520 #>>41844782 #>>41847696 #
4. hidelooktropic ◴[15 Oct 24 03:06 UTC] No.41844520{3}[source]▶
To run gguf files? LM Studio for one. I think recurse on macos as well and probably some others.
replies(2): >>41847312 #>>41853147 #
5. kristianp ◴[15 Oct 24 03:31 UTC] No.41844635[source]▶
It seems that zamba 2 isn't supported yet, the previous model's issue is here:

Feature Request: Support Zyphra/Zamba2-2.7B #8795

Open tomasmcm opened this issue on Jul 31 · 1 comment

https://github.com/ggerganov/llama.cpp/issues/8795

6. gbickford ◴[15 Oct 24 03:53 UTC] No.41844782{3}[source]▶
If you look in the `config.json`[1] it shows `Zamba2ForCausalLM`. You can use a version of the transformers library to do inference that supports that.

The model card states that you have to use their fork of transformers.[2]

1. https://huggingface.co/Zyphra/Zamba2-7B-Instruct/blob/main/c...

2. https://huggingface.co/Zyphra/Zamba2-7B-Instruct#prerequisit...

7. x_may ◴[15 Oct 24 11:10 UTC] No.41847312{4}[source]▶
As another commenter said, this has no GGUF because it’s partially mamba based which is unsupported in llama.cpp
8. Havoc ◴[15 Oct 24 12:01 UTC] No.41847638[source]▶
Mamba based stuff tends to take longer to become available
9. wazoox ◴[15 Oct 24 12:09 UTC] No.41847696{3}[source]▶
Gpt4all is a good and easy way to run gguf models.
10. xyc ◴[15 Oct 24 21:08 UTC] No.41853147{4}[source]▶
dev of https://recurse.chat/ here, thanks for mentioning! rn we are focusing on features like shortcuts/floating window, but will look into support this in some time. to add to the llama.cpp support discussion, it's also worth noting that llama.cpp does not yet support gpu for mamba models https://github.com/ggerganov/llama.cpp/issues/6758