Back in the days of commodities based currency, the demand for silver or gold limited what could be achieved in the economy. After a flood of silver stolen from the "New World" arrived in Europe, it caused inflation. Similar situations happen throughout history.
I think the primary driver of economic growth is the distribution of opportunity throughout society. If a rentier class happens to capture governance, as is the case in Russia with the oligarchs, or in the US with our "Tech Bros" and hedge funds, social mobility, and the drive to improve one's station in life, is greatly diminished, and the economy becomes a zero sum game.
I'm very open to being wrong though.