←back to thread

Can LLMs accurately recall the Bible?

(benkaiser.dev)
223 points benkaiser | 2 comments | 29 Dec 24 03:44 UTC | HN request time: 0s | source
Show context
michaelsbradley ◴[29 Dec 24 05:49 UTC] No.42537899[source]▶
I’ve been pretty impressed with ChatGPT’s promising capabilities as a research assistant/springboard for complex inquiries into the Bible and patristics. Just one example:

   Can you provide short excerpts from works in Latin and Greek written between 600 and 1300 that demonstrate the evolution over those centuries specifically of literary references to Jesus' miracle of the loaves and fishes?
https://chatgpt.com/share/675858d5-e584-8011-a4e9-2c9d2df783...
replies(3): >>42538262 #>>42538286 #>>42538320 #
1. edflsafoiewq ◴[29 Dec 24 07:09 UTC] No.42538262[source]▶
How certain are you that's correct? IME these "search problems" are the kind of thing almost always provokes hallucinations.

For example, I looked up the quotation provided from Isidore of Seville's De fide catholica contra Iudaeos, Lib. II, cap. 19, using this copy on WikiSource, https://la.wikisource.org/wiki/De_fide_catholica_contra_Iuda.... The quote certainly does not appear under LIBER SECUNDUS, CAPUT XIX. Nor could I find it in whole or in fragment anywhere in the document, nor indeed any mention of the miracle of loaves and fishes (granted, I could have missed one, I relied on Ctrl+F and my very rusty Latin).

Perhaps the copy on WikiSource is incomplete, or perhaps there are differing manuscripts, but perhaps also the quote was a complete hallucination to begin with.

replies(1): >>42538305 #
2. FearNotDaniel ◴[29 Dec 24 07:21 UTC] No.42538305[source]▶
Exactly - it’s the same problem when using (current) LLMs for major programming tasks, generally useless if you don’t already have enough knowledge of the language/platform to spot and correct the mistakes, plus enough awareness of software design and architecture to recognise what is going to be secure, performant and maintainable in the long run.