←back to thread

Adaptive LLM routing under budget constraints

(arxiv.org)
204 points tdchaitanya | 1 comments | 01 Sep 25 16:57 UTC | HN request time: 0s | source
Show context
pbd ◴[01 Sep 25 17:49 UTC] No.45094941[source]▶
GPT-4 at $24.7 per million tokens vs Mixtral at $0.24 - that's a 100x cost difference! Even if routing gets it wrong 20% of the time, the economics still work. But the real question is how you measure 'performance' - user satisfaction doesn't always correlate with technical metrics.
replies(5): >>45095081 #>>45095225 #>>45095267 #>>45095811 #>>45095813 #
pqtyw ◴[01 Sep 25 18:20 UTC] No.45095225[source]▶
> GPT-4 at $24.7 per million tokens

While technically true why would you want to use it when OpenAI itself provides a bunch of many times cheaper and better models?

replies(1): >>45095604 #
1. KTibow ◴[01 Sep 25 19:04 UTC] No.45095604[source]▶
RouterBench is from March 2024.