Learn Prolog Now

I am once again shilling the idea that someone should find a way to glue Prolog and LLMs together for better reasoning agents.

https://news.ycombinator.com/context?id=43948657

Thesis:

1. LLMs are bad at counting the number of r's in strawberry.

2. LLMs are good at writing code that counts letters in a string.

3. LLMs are bad at solving reasoning problems.

4. Prolog is good at solving reasoning problems.

5. ???

6. LLMs are good at writing prolog that solves reasoning problems.

Common replies:

1. The bitter lesson.

2. There are better solvers, ex. Z3.

3. Someone smart must have already tried and ruled it out.

Successful experiments:

1. https://quantumprolog.sgml.net/llm-demo/part1.html

> "4. Prolog is good at solving reasoning problems."

Plain Prolog's way of solving reasoning problems is effectively:

    for person in [martha, brian, sarah, tyrone]:
      if timmy.parent == person:
        print "solved!"

You hard code some options, write a logical condition with placeholders, and Prolog brute-forces every option in every placeholder. It doesn't do reasoning.

Arguably it lets a human express reasoning problems better than other languages by letting you write high level code in a declarative way, instead of allocating memory and choosing data types and initializing linked lists and so on, so you can focus on the reasoning, but that is no benefit to an LLM which can output any language as easily as any other. And that might have been nice compared to Pascal in 1975, it's not so different to modern garbage collected high level scripting languages. Arguably Python or JavaScript will benefit an LLM most because there are so many training examples inside it, compared to almost any other langauge.

What makes you think your brain isn't also brute forcing potential solutions subconciously and only surfacing the useful results?

human brains are insanely powerful pattern matching and shortcut-taking machines. There's very little brute forcing going on.