←back to thread

Every Way to Get Structured Output from LLMs

(www.boundaryml.com)
169 points constantinum | 3 comments | 18 Jun 24 04:01 UTC | HN request time: 1.042s | source
Show context
StrauXX ◴[18 Jun 24 07:03 UTC] No.40714872[source]▶
Did I understand the documentation for many of these libraries correctly in that they reprompt until they receive valid JSON? If so I don't understand why one would do that when token masking is a deterministicly verifyable way to get structured output of any kind (as done by Guidance and LMQL for instance). This is not meant to be snarky, I really am curious. Is there an upside to reprompting - aside from easier implementation.
replies(4): >>40714984 #>>40714988 #>>40715185 #>>40715620 #
1. hellovai ◴[18 Jun 24 07:21 UTC] No.40714984[source]▶
the main one is that most people don't own the model. so if you use openai / anthropic / etc then you can't use token masking. in that case, reprompting is pretty much the only option
replies(2): >>40716262 #>>40725394 #
2. michaelt ◴[18 Jun 24 11:02 UTC] No.40716262[source]▶
In the specific cases of openai and anthropic, both have 'tool use' interfaces which will generate valid JSON following a schema of your choice.

You're right, though, that reprompting works with pretty much everything out there, including hosted models that don't have tool use as part of their API. And its simple too, you don't even need to know what "token masking" is.

Reprompting can also apply arbitrarily criteria that are more complex than just a json schema. You ask it to choose an excerpt of a document and the string it returns isn't an excerpt? Just reprompt.

3. StrauXX ◴[19 Jun 24 06:21 UTC] No.40725394[source]▶
It does. With OpenAI at least you definetly can use token masking. There are some limitations but even those are circumventable. I have used token masking on the OpenAI API with LMQL without any issues.