I built a prototype version similar to this called PastPort last year, but I like your idea better.
This uses Flux then image to video? Good quality generations, it would be wonderful to see the accuracy of the images improve. I saw you want to make it interactive like moving mode in geoguessr; that would be fantastic. I can imagine a few ways to do both.
I don't believe this is open source - is there a way to contribute to this? One man operation or are you a team?