←back to thread

268 points Areibman | 1 comments | | HN request time: 0.219s | source

Hey HN! Tokencost is a utility library for estimating LLM costs. There are hundreds of different models now, and they all have their own pricing schemes. It’s difficult to keep up with the pricing changes, and it’s even more difficult to estimate how much your prompts and completions will cost until you see the bill.

Tokencost works by counting the number of tokens in prompt and completion messages and multiplying that number by the corresponding model cost. Under the hood, it’s really just a simple cost dictionary and some utility functions for getting the prices right. It also accounts for different tokenizers and float precision errors.

Surprisingly, most model providers don't actually report how much you spend until your bills arrive. We built Tokencost internally at AgentOps to help users track agent spend, and we decided to open source it to help developers avoid nasty bills.

Show context
Lerc ◴[] No.40711341[source]
With all the options there seems like an opportunity for a single point API that can take a series of prompts, a budget and a quality hint to distribute batches for most bang for buck.

Maybe a small triage AI to decide how effectively models handle certain prompts to preserve spending for the difficult tasks.

Does anything like this exist yet?

replies(3): >>40712921 #>>40715521 #>>40715879 #
curious_cat_163 ◴[] No.40712921[source]
I have yet to find a use case where quality can be traded off.

Would love to hear what you had in mind.

replies(3): >>40713311 #>>40713472 #>>40788282 #
1. Breza ◴[] No.40788282[source]
I've encountered plenty of tasks where lower quality models work quite well. I prefer using Claude 3 Opus, DBRX, or Llama-3, but that level of quality isn't always needed. Here are a few examples.

Top story picker. Given a bunch of news stories, pick which one should be the lead story.

Data viz color picker. Given a list of categories for a chart, return a color for each one.

Windows Start menu. Given a list of installed programs and a query, select the five most likely programs that the user wants.