←back to thread

Can LLMs accurately recall the Bible?

(benkaiser.dev)
223 points benkaiser | 2 comments | 29 Dec 24 03:44 UTC | HN request time: 0s | source
Show context
michaelsbradley ◴[29 Dec 24 05:49 UTC] No.42537899[source]▶
I’ve been pretty impressed with ChatGPT’s promising capabilities as a research assistant/springboard for complex inquiries into the Bible and patristics. Just one example:

   Can you provide short excerpts from works in Latin and Greek written between 600 and 1300 that demonstrate the evolution over those centuries specifically of literary references to Jesus' miracle of the loaves and fishes?
https://chatgpt.com/share/675858d5-e584-8011-a4e9-2c9d2df783...
replies(3): >>42538262 #>>42538286 #>>42538320 #
FearNotDaniel ◴[29 Dec 24 07:16 UTC] No.42538286[source]▶
I am by no means a professional in this area, but as a keen amateur I would worry about my inability to discern facts from hallucinations in such a scenario: while I could imagine such output provides a useful “springboard” set of references for someone already skilled in the right area, without being able to look up the original texts myself and make sense of the Latin/Greek I would not feel confident that such texts even really exist, let alone if they contain the actual words the LLM claims and if the translations are any good. And that’s before you get into questions of the “status” of any given work (was it considered accurate or apocryphal at the time of writing, for which audience was it intended using what kind of literary devices, what if any is the modern scholarly consensus on the value, truth or legitimacy of the text etc etc)
replies(1): >>42538318 #
1. wizzwizz4 ◴[29 Dec 24 07:25 UTC] No.42538318[source]▶
> without being able to look up the original texts myself

Rule of thumb: if you can't look up the original texts, you can assume they weren't actually in the training data. The training data is, however, likely to include a lot of people quoting those texts, meaning that the model predicts "SOURCE says OPEN QUOTATION MARK" and then tries to autocomplete it. If you can verify it, you might not need to; but if you can't verify it, it's certainly wrong.

replies(1): >>42563955 #
2. nickpsecurity ◴[01 Jan 25 04:28 UTC] No.42563955[source]▶
"Rule of thumb: if you can't look up the original texts, you can assume they weren't actually in the training data. "

That's not reliable. I've found them on the Internet in various forms (eg studybible.info). Google Books also has scanned copies of many, ancient writings. There's probably obscure sites people would miss. If searching for them, the search algorithms might avoid them to instead prioritize newer, click-bait content.

Telling what wasn't in the training data for sure should be considered impossible right now. If it matters, we need to use models with open, legal-to-share, training data. If that's impossible, one might at least use a model with training data accessible to them (eg free + licensed).