Can LLMs accurately recall the Bible?

I tested this back when GPT4 was new. I found ChatGPT could quote the verses well. If I asked it to summarize something, it would sometimes hallucinate stuff that had nothing to do with what was in the text. If I prompted it carefully, it could do a proper exegesis of many passages using the historical-grammatical method.

I believe this happens because the verses and verse-specific commentary are abundant in the pre-training sources they used. Whereas, if one asks a highly-interpretive question, then it starts re-hashing other patterns in its training data which are un-Biblical. Asking about intelligent design, it got super hostile trying to beat me into submission to its materialistic worldview every paragraph.

So, they have their uses. I’ve often pushed for a large model trained on Project Gutenberg to have a 100% legal model for research and personal use. A side benefit of such a scheme would be that Gutenberg has both Bibles and good commentaries which trainers could repeat for memorization. One could add licensed, Christian works on a variety of topics to a derived model to make a Christian assistant AI.