Every Way to Get Structured Output from LLMs

Did I understand the documentation for many of these libraries correctly in that they reprompt until they receive valid JSON? If so I don't understand why one would do that when token masking is a deterministicly verifyable way to get structured output of any kind (as done by Guidance and LMQL for instance). This is not meant to be snarky, I really am curious. Is there an upside to reprompting - aside from easier implementation.

My experience with models even about a year ago is that the model has firmly decided what it thinks by the time of the last layer, so the probabilities on that layer aren't very useful.

You either get the same (in this case wrong) thing differently worded, or worse you get effectively noise if the second probability is very much lower than the largest probability.

My guess is that applies here too. Better to let all the layers rethink the tokens, than force hallucination of eg a random letter when you don't expect an angle bracket

(Edit: above is assuming using logprobs and/or logit_bias with the OpenAI API, not some other masking technique)