AI noob question:
Why do OpenAI/Anthropic/... not support constraining token generation? I'd imagine producing valid structured output would be at the top of their feature request lists.
replies(3):
Why do OpenAI/Anthropic/... not support constraining token generation? I'd imagine producing valid structured output would be at the top of their feature request lists.
client.chat.completions.create(..., response_format={"type": "json_object"})
But the nature of LLMs is stochastic, nothing is 100%. The LLM vendors aren't dummies and train hard for this use case. But you still need a prompt that OpenAI can handle, and validating / fixing the output with an output parser, and retrying.
In my experience asking for simple stuff, requesting json_object is reliable.
with LangChain even! eye-roll, you can't really title the post 'every way' and omit possibly the most popular way with a weak dig. I have literally no idea why they would omit it, it's just a thin wrapper over the LLM APIs and has a JSON output parser. Of course people do use LangChain in production, although there is merit to the idea of using it for research, trying different LLMs and patterns where LangChain makes it easy to try different things, and then using the underlying LLM directly in prod which will have a more stable API and fewer hinky layers.
this post is a little frustrating since it doesn't explain things that a dev would want to know, and omits the popular modules. the comment by resiros offers some good additional info.