(qwenlm.github.io)

114 points cmcconomy | 1 comments | 18 Nov 24 16:27 UTC | HN request time: 0s | source

Show context

lr1970 ◴[19 Nov 24 00:41 UTC] No.42178968[source]▶

> We have extended the model’s context length from 128k to 1M, which is approximately 1 million English words

Actually English language tokenizers map on average 3 words into 4 tokens. Hence 1M tokens is about 750K English words not a million as claimed.

1. ◴[19 Nov 24 01:05 UTC] No.42179102[source]▶

Extending the context length to 1M tokens