Every Way to Get Structured Output from LLMs

The baml config files look a lot like code. For example in baml:

    class Resume {
        name string
        education Education[] @description("Extract in the same order listed")
        skills string[] @description("Only include programming languages")
    }

Could be expressed in Python like this:

    class Resume:
        name: str
        education: List[Education] # Extract in the same order listed
        skills: List[str] # Only include programming languages

Two benefits I see are that it would make the file leaner (because Python is nicely lean) and provide free parsing and syntax highlighting.

Is there a benefit of rolling your own DSL?

that's a great question, there's three main benefits:

1. seeing the full prompt, even though that python code feels leaner, somehow you need to convert it to a prompt. a library will do that in some way, BAML has a VSCode playground to see the entire prompt + tokenization. If we had to do this off of python/ts, we would run into the halting problem and making the playground would be much much harder.

2. there's a lot of codegen we do for users, to make life easier, e.g. w/o BAML, to now do streaming for the resume, you would have to do something like this:

class PartialResume: name: Optional[str] education: List[PartialEducation] skills: List[str]

and then at some point you need to reparse PartialResume -> Resume, we can codegen all of that for you, and give you autocomplete, type-safety for free.

3. We added a lot of static analysis / jump to definition etc to JINJA (which we use for strings), and that is much easier to navigate than f-strings.

4. Since its code-gen we can support all languages way easier, so prompting techniques in python work the same exact way for the same code in typescript.

Two big advantages.

First, if you want a declarative config with limited, domain-specific options, rolling your own DSL instead of using something as complex as Python is much, much easier to implement. You're not actually going to be running the code either way, at least not in the normal way, and the Python syntax tree is pretty complicated.

Second, having code that looks like Python can lead your users to believe that it is in, in fact, Python. When you're doing things like using your DSL as configuration that happens at setup time, but then actually "running" the resulting config later on, that can lead to people getting themselves into trouble - for instance, they might try to use `time.now()` and end up embedding the time of the config parser as a constant in their workflow definition.

If you want to use Python as your language, you probably want to define your "DSL" as a Python library, so that you can use a normal interpreter to work with it. Maybe you have library functions that return config objects, and a user's "configuration" is an arbitrary Python file with a standard function name as an entry point. But then when you want to introspect over types, you probably need to start playing games with decorators, which is tricky again, and you have to be very careful to have that evaluation step return meaningful errors.

Starlark (https://github.com/bazelbuild/starlark) is an example of using Python-ish as a "configuration" language. That took an absolutely massive amount of engineering to get to be well-defined, and was only arguably worth it because they wanted a language that's a loop construct away from being Turing-complete. If they had wanted a basic declarative relationship language, they probably would have used textprotos or GCL.

Using Pydantic also looks very close to the DSL (trivial to translate mechanically)

https://docs.pydantic.dev/latest/concepts/models/#dynamic-mo...