←back to thread

Adaptive LLM routing under budget constraints

(arxiv.org)
204 points tdchaitanya | 1 comments | 01 Sep 25 16:57 UTC | HN request time: 0.202s | source
Show context
pbd ◴[01 Sep 25 17:49 UTC] No.45094941[source]▶
GPT-4 at $24.7 per million tokens vs Mixtral at $0.24 - that's a 100x cost difference! Even if routing gets it wrong 20% of the time, the economics still work. But the real question is how you measure 'performance' - user satisfaction doesn't always correlate with technical metrics.
replies(5): >>45095081 #>>45095225 #>>45095267 #>>45095811 #>>45095813 #
FINDarkside ◴[01 Sep 25 18:26 UTC] No.45095267[source]▶
It's trivial to get better score than GPT-4 with 1% of the cost by using my propertiary routing algorithm that routes all requests to Gemini 2.5 Flash. It's called GASP (Gemini Always, Save Pennies)
replies(1): >>45095832 #
nutjob2 ◴[01 Sep 25 19:30 UTC] No.45095832[source]▶
Does anyone working in an individual capacity actually end up paying for Gemini (Flash or Pro)? Or does Google boil you like a frog and you end up subscribing?
replies(4): >>45095961 #>>45096427 #>>45099635 #>>45100282 #
aspect8445 ◴[01 Sep 25 19:45 UTC] No.45095961[source]▶
I've used Gemini in a lot of personal projects. At this point I've probably made tens of thousands of requests, sometimes exceeding 1k per week. So far, I haven't had to pay a dime!
replies(1): >>45097258 #
worm00111 ◴[01 Sep 25 22:43 UTC] No.45097258[source]▶
How come you don't need to pay? Do you get it for free somehow?
replies(1): >>45097410 #
KETHERCORTEX ◴[01 Sep 25 23:09 UTC] No.45097410[source]▶
There's free tier for API.
replies(1): >>45098785 #
1. drittich ◴[02 Sep 25 03:14 UTC] No.45098785[source]▶
"When you use Unpaid Services, including, for example, Google AI Studio and the unpaid quota on Gemini API, Google uses the content you submit to the Services and any generated responses to provide, improve, and develop Google products and services and machine learning technologies, including Google's enterprise features, products, and services, consistent with our Privacy Policy.

To help with quality and improve our products, human reviewers may read, annotate, and process your API input and output. Google takes steps to protect your privacy as part of this process. This includes disconnecting this data from your Google Account, API key, and Cloud project before reviewers see or annotate it. Do not submit sensitive, confidential, or personal information to the Unpaid Services."

Reference: https://ai.google.dev/gemini-api/terms