←back to thread

Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

(github.com)
279 points matthewolfe | 1 comments | 30 Jun 25 12:33 UTC | HN request time: 0s | source

TokenDagger is a drop-in replacement for OpenAI’s Tiktoken (the tokenizer behind Llama 3, Mistral, GPT-3.*, etc.). It’s written in C++ 17 with thin Python bindings, keeps the exact same BPE vocab/special-token rules, and focuses on raw speed.

I’m teaching myself LLM internals by re-implementing the stack from first principles. Profiling TikToken’s Python/Rust implementation showed a lot of time was spent doing regex matching. Most of my perf gains come from a) using a faster jit-compiled regex engine; and b) simplifying the algorithm to forego regex matching special tokens at all.

Benchmarking code is included. Notable results show: - 4x faster code sample tokenization on a single thread. - 2-3x higher throughput when tested on a 1GB natural language text file.

Show context
superlopuh ◴[30 Jun 25 17:27 UTC] No.44425827[source]▶
Can someone familiar with performance of LLMs please tell me how important this is to the overall perf? I'm interested in looking into optimizing tokenizers, and have not yet run the measurements. I would have assumed that the cost is generally dominated by matmuls but am encouraged by the reception of this post in the comments.
replies(4): >>44426006 #>>44426054 #>>44427256 #>>44428487 #
serjester ◴[30 Jun 25 17:44 UTC] No.44426006[source]▶
Tokenizing text is ridiculously small part of the overall computation that goes into serving a request. With that said if you’re doing this on petabytes of data, never hurts to have something faster.
replies(1): >>44426605 #
1. odyssey7 ◴[30 Jun 25 18:53 UTC] No.44426605[source]▶
A language that isn’t memory-safe can definitely hurt. AI needs more security, not less.