←back to thread

Every Way to Get Structured Output from LLMs

(www.boundaryml.com)
169 points constantinum | 1 comments | 18 Jun 24 04:01 UTC | HN request time: 0.204s | source
Show context
mg ◴[18 Jun 24 07:23 UTC] No.40714992[source]▶
The baml config files look a lot like code. For example in baml:

    class Resume {
        name string
        education Education[] @description("Extract in the same order listed")
        skills string[] @description("Only include programming languages")
    }
Could be expressed in Python like this:

    class Resume:
        name: str
        education: List[Education] # Extract in the same order listed
        skills: List[str] # Only include programming languages
Two benefits I see are that it would make the file leaner (because Python is nicely lean) and provide free parsing and syntax highlighting.

Is there a benefit of rolling your own DSL?

replies(3): >>40715066 #>>40715071 #>>40715322 #
1. GeneralMayhem ◴[18 Jun 24 07:37 UTC] No.40715071[source]▶
Two big advantages.

First, if you want a declarative config with limited, domain-specific options, rolling your own DSL instead of using something as complex as Python is much, much easier to implement. You're not actually going to be running the code either way, at least not in the normal way, and the Python syntax tree is pretty complicated.

Second, having code that looks like Python can lead your users to believe that it is in, in fact, Python. When you're doing things like using your DSL as configuration that happens at setup time, but then actually "running" the resulting config later on, that can lead to people getting themselves into trouble - for instance, they might try to use `time.now()` and end up embedding the time of the config parser as a constant in their workflow definition.

If you want to use Python as your language, you probably want to define your "DSL" as a Python library, so that you can use a normal interpreter to work with it. Maybe you have library functions that return config objects, and a user's "configuration" is an arbitrary Python file with a standard function name as an entry point. But then when you want to introspect over types, you probably need to start playing games with decorators, which is tricky again, and you have to be very careful to have that evaluation step return meaningful errors.

Starlark (https://github.com/bazelbuild/starlark) is an example of using Python-ish as a "configuration" language. That took an absolutely massive amount of engineering to get to be well-defined, and was only arguably worth it because they wanted a language that's a loop construct away from being Turing-complete. If they had wanted a basic declarative relationship language, they probably would have used textprotos or GCL.