Most active commenters
  • simonw(3)

←back to thread

Show HN: Token price calculator for 400+ LLMs

(github.com)
268 points Areibman | 19 comments | 17 Jun 24 19:44 UTC | HN request time: 0.217s | source | bottom

Hey HN! Tokencost is a utility library for estimating LLM costs. There are hundreds of different models now, and they all have their own pricing schemes. It’s difficult to keep up with the pricing changes, and it’s even more difficult to estimate how much your prompts and completions will cost until you see the bill.

Tokencost works by counting the number of tokens in prompt and completion messages and multiplying that number by the corresponding model cost. Under the hood, it’s really just a simple cost dictionary and some utility functions for getting the prices right. It also accounts for different tokenizers and float precision errors.

Surprisingly, most model providers don't actually report how much you spend until your bills arrive. We built Tokencost internally at AgentOps to help users track agent spend, and we decided to open source it to help developers avoid nasty bills.

1. simonw ◴[17 Jun 24 20:47 UTC] No.40710871[source]▶
I don't understand how the Claude functionality works.

As far as I know Anthropic haven't released the tokenizer for Claude - unlike OpenAI's tiktoken - but your tool lists the Claude 3 models as supported. How are you counting tokens for those?

replies(3): >>40710980 #>>40711374 #>>40718095 #
2. dudeinhawaii ◴[17 Jun 24 20:57 UTC] No.40710980[source]▶
It's open source so you can take a look (I'm not the author): https://github.com/AgentOps-AI/tokencost/blob/main/tokencost...

It looks like tiktoken is the default for most of the methods.

Disclaimer: I didn't fully trace which are being used in each case/model.

replies(2): >>40711095 #>>40711161 #
3. refibrillator ◴[17 Jun 24 21:09 UTC] No.40711095[source]▶
> # TODO: Add Claude support

There are no cases for Claude models yet.

I wonder if anyone has run a bunch of messages through Anthropic's API and used the returned token count to approximate the tokenizer?

replies(1): >>40718364 #
4. simonw ◴[17 Jun 24 21:16 UTC] No.40711161[source]▶
Yeah, I asked here because I dug around in the code and couldn't see how they were doing this, wanted to check I hadn't missed something.
5. Areibman ◴[17 Jun 24 21:40 UTC] No.40711374[source]▶
Anthropic actually has a Claude 3 tokenizer tucked away in one of their repos: https://github.com/anthropics/anthropic-tokenizer-typescript

At this moment, Tokencost uses the OpenAI tokenizer as a default tokenizer, but this would be a welcome PR!

replies(1): >>40711504 #
6. simonw ◴[17 Jun 24 21:53 UTC] No.40711504[source]▶
"This package can be used to count tokens for Anthropic's older models. As of the Claude 3 models, this algorithm is no longer accurate [...]"

I've been bugging Anthropic about this for a while, they said that releasing a new tokenizer is not on their current roadmap.

replies(1): >>40711734 #
7. throwaway211 ◴[17 Jun 24 22:14 UTC] No.40711734{3}[source]▶
Imagine a coffee shop refusing to have a price list until after the coffee's been made.
replies(3): >>40712333 #>>40715060 #>>40717521 #
8. kevindamm ◴[17 Jun 24 23:20 UTC] No.40712333{4}[source]▶
In many countries a taxi won't tell you how much the ride will cost. The first time I traveled to somewhere that negotiated the cost up front it blew my mind.

Frequently, contracts will have room for additional charges if circumstances change even a little, or products will have a market rate (fish, equity, etc.).

It might seem absurd but variable cost things are not uncommon.

replies(5): >>40712570 #>>40714914 #>>40715972 #>>40720216 #>>40777992 #
9. MattGaiser ◴[17 Jun 24 23:49 UTC] No.40712570{5}[source]▶
In Oman, every tax fare was basically just of the bills. So 5 rials, 10 rials, or 20 rials. So many different ways people price things around the world.
10. _flux ◴[18 Jun 24 07:09 UTC] No.40714914{5}[source]▶
In this case there's nothing that's variable, though, and the competition is able to pull it off precisely. Indeed, they themselves were able to do it before!
replies(1): >>40727334 #
11. itake ◴[18 Jun 24 07:36 UTC] No.40715060{4}[source]▶
reminds me of the coffee shop incident in seattle last week with the hammer
12. scoot ◴[18 Jun 24 10:19 UTC] No.40715972{5}[source]▶
> In many countries a taxi won't tell you how much the ride will cost.

I've only ever seen: fixed price based on destination (typically for fares originating from an airport), negotiated, or metered. A better analog analogy would be metered pricing, but where the cost per mile is a secret.

replies(1): >>40749171 #
13. amanda99 ◴[18 Jun 24 13:19 UTC] No.40717521{4}[source]▶
I mean you already pay per output token and (sure, you can limit it), but it's unpredictable given the prompt?
14. J_Shelby_J ◴[18 Jun 24 14:11 UTC] No.40718095[source]▶
Here you go https://github.com/javirandor/anthropic-tokenizer
15. GranPC ◴[18 Jun 24 14:38 UTC] No.40718364{3}[source]▶
Sorta: https://github.com/javirandor/anthropic-tokenizer
16. szundi ◴[18 Jun 24 17:35 UTC] No.40720216{5}[source]▶
These are called the variable variable costs now
17. kevindamm ◴[19 Jun 24 11:46 UTC] No.40727334{6}[source]▶
Perhaps they have a new tokenization method that's non-deterministic? If there are parallel lookaheads, not necessarily an rng, race conditions could make for variable cost. Or an expansion of certain terms into multiple token outputs, but the selection of which terms are expanded is based on a dynamic pool of Named Entities being recognized. Or maybe they just want to hide their process.. there is some secret sauce there even if it's deterministic, and so much depends on getting a good initial embedding, I've seen a tokenizer make or break an otherwise great model.

I am merely hypothesizing, it may not be nondeterministic but I'm not going to assume it's not.

18. sitkack ◴[21 Jun 24 13:02 UTC] No.40749171{6}[source]▶
Where the cost per mile is published, but their mile ain’t yer mile.
19. Breza ◴[24 Jun 24 16:26 UTC] No.40777992{5}[source]▶
This is what Washington DC did when I moved here. They theoretically had zones, but in reality it was arbitrary. Moving to meters was an amazing development.

Similarly, as LLMs become more and more commonplace, the pricing models will need to be more predictable. My LLM expenses are only around $100/month, but it's a bigger impediment to pushing projects to production when I can't tell the boss exactly how it'll be priced.