I understand the audiobook server, but what’s the use case for the podcast part? You replicate a podcast on your own server, in case the original goes away?
Some podcast feeds only list the last N episodes, so if you want to listen to episode N+1, you either have to have it already downloaded locally or cache the feed and hope the audio file's URL is valid when you go to listen to it later.