I think they simply did what AAA video games did. They found what sold best at one moment in time and then obsessively tried to work to copy that.
But the problem is that people don't want to play 40 different Call of Duties, or watch 30 different Batmen. It's just that Batman or Call of Duty were the 'meet in the middle' of a variety of different tastes. But when those other tastes aren't accounted for, it becomes nauseating. It's like how most of everybody really likes cake icing, but eating nothing but cake icing is quite a repulsive concept.
I think things like Dune, Interstellar, and other such films emphasize that there's a gaping hole in the market for things besides men in spandex, but it's just not being filled. And there's even extensive social commentary in Dune (as in the book) but it's done through metaphor rather than shoving it down your throat. And the movie is also rather slow paced with some 3 key events playing out in a 155 minute film, yet it continues to do extremely well. On the other hand those Fremen suits are kind of spandexy...