> I don't know of a single person who has switched to a smaller phone after having a bigger one
Consider how small the overlap is between devices in a product lineup in the first place and an audience that can buy them -- whether that's through carrier availability to put them on plans if in the US, or the resources to spend on them up-front.
That trend also is based on touch as the primary method of interaction -- but given the tethered AR devices we're starting to see trickle out, and Android's desktop mode finally hitting prime time, that assumption might not hold long-term. I'm not saying this will be the timeline we live on, but considering some of the experiments with dedicated, external devices for powering them, it's not hard to envision the pendulum swinging back toward smaller phones that focus more on things like the compute and sensors and less around a screen you look at all day.
Think of the (modern) Moto Razr. You could, hypothetically, have a compute device that more so resembles the folded-down version of this -- aimed more toward external displays, and less toward being regularly looked at.