Ask HN: Any active COBOL devs here? What are you working on?

I'm sure they can do brainfuck if you have a good training set.

LLMs are terrible at brainfuck. I spent a solid week attempting to use generative models to iteratively refine BF program tapes with nothing to show for it. I've written genetic programming routines that can produce better brainfuck programs than ChatGPT can.

For example, if I prompt ChatGPT: "Write me a BF program that produces the alphabet, but inverts the position of J & K" it will deterministically fail. I've never even seen one that produces the alphabet the normal way. I can run a GP algorithm over an example of the altered alphabet string and use simple MSE to get it to evolve a BF program that actually emits the expected output.

The BPE tokenizer seems like a big part of the problem when considering the byte-per-instruction model, but fundamentally I don't think there is a happy path even if we didn't need to tokenize the corpus. The expressiveness of the language is virtually non-existent. Namespaces, type names, member names, attributes, etc., are a huge part of what allows for a LLM to lock on to the desired outcome. Getting even one byte wrong is catastrophic for the program's meaning. You can get a lot of bytes wrong in C/C++/C#/Java/Go/etc. (e.g. member names) and still have the function do exactly the same thing.

That was a very serious response to an off-the-cuff joke.

BUT: Please, oh please, write up a blog entry! I bet that would be fun to read.