←back to thread

Extending the context length to 1M tokens

(qwenlm.github.io)
114 points cmcconomy | 3 comments | 18 Nov 24 16:27 UTC | HN request time: 0.503s | source
Show context
anon291 ◴[18 Nov 24 17:49 UTC] No.42174879[source]▶
Can we all agree that these models far surpass human intelligence now? I mean they process hours worth of audio in less time than it would take a human to even listen. I think the singularity passed and we didn't even notice (which would be expected)
replies(11): >>42174949 #>>42174987 #>>42175002 #>>42175008 #>>42175019 #>>42175095 #>>42175118 #>>42175171 #>>42175223 #>>42175324 #>>42176838 #
1. nisarg2 ◴[18 Nov 24 18:14 UTC] No.42175171[source]▶
At the end of the response they forget everything. They need to be fed the entire text for them to know anything about it the next time. That is not surpassing even feline intelligence.
replies(2): >>42175217 #>>42175305 #
2. andai ◴[18 Nov 24 18:17 UTC] No.42175217[source]▶
A genius can have anterograde amnesia and still be a genius.
3. anon291 ◴[18 Nov 24 18:23 UTC] No.42175305[source]▶
If we did to cats what he did to GPT models that would be animal abuse.

That is to say, if we want to extend this analogy, the model is 'killed' after each round. This is hardly a criticism of the underlying technology.

Going back to feeding the entire input. That is not really true. There are a dozen ways to not do that these day.