←back to thread

Extending the context length to 1M tokens

(qwenlm.github.io)
114 points cmcconomy | 1 comments | 18 Nov 24 16:27 UTC | HN request time: 0s | source
Show context
aliljet ◴[18 Nov 24 18:05 UTC] No.42175062[source]▶
This is fantastic news. I've been using Qwen2.5-Coder-32B-Instruct with Ollama locally and it's honestly such a breathe of fresh air. I wonder if any of you have had a moment to try this newer context length locally?

BTW, I fail to effectively run this on my 2080 ti, I've just loaded up the machine with classic RAM. It's not going to win any races, but as they say, it's not the speed that matter, it's the quality of the effort.

replies(3): >>42175226 #>>42176314 #>>42177831 #
notjulianjaynes ◴[18 Nov 24 18:17 UTC] No.42175226[source]▶
Hi, are you able to use Qwen's 128k context length with Ollama? Using AnythingLLM + Ollamma and a GGUF version I kept getting an error message with prompts longer than 32,000 tokens. (summarizing long transcripts)
replies(1): >>42175335 #
syntaxing ◴[18 Nov 24 18:25 UTC] No.42175335[source]▶
The famous Daniel Chen (same person that made Unsloth and fixed Gemini/LLaMa bugs) mentioned something about this on reddit and offered a fix. https://www.reddit.com/r/LocalLLaMA/comments/1gpw8ls/bug_fix...
replies(2): >>42175727 #>>42175742 #
1. notjulianjaynes ◴[18 Nov 24 19:05 UTC] No.42175742[source]▶
Yeah unfortunately that's the exact model I'm using (Q5 version. What I've been doing is first loading the transcript into the vector database, and then giving it a prompt thats like "summarize the transcript below: <full text of transcript>". This works surprisingly well except for one transcript I had which was of a 3 hour meeting that was per an online calculator about 38,000 tokens. Cutting the text up into 3 parts and pretending each was a seperate meeting* lead to a bunch of hallucinations for some reason.

*In theory this shouldn't matter much for my purpose of summarizing city council meetings that follow a predictable format.