←back to thread

261 points fzliu | 1 comments | | HN request time: 0.209s | source
1. unit149 ◴[] No.42163393[source]
In the traditional Python API, the Voyage engine will tokenize blocks of text and output a string of characters. This model seems to be doing that by vectorizing images in space.

Words like 'you' and 'apple' will be a unitary token. More complex terms like 'pikachu' may be divided into pik-a-chu.

[1]: https://docs.voyageai.com/docs/tokenization