I think that's right. At some level, any anticipation of a future state has to be measurable in some kind of confidence level.
I suppose where I get lost is that, at least subjectively, I end up treating different anticipated return distributions differently. I want a mixture of risks, both in terms of their covariance and the absolute properties.
When I think of "speculation", I am intentionally shaping only a small portion of my personal portfolio toward high risk, high reward activities. And I only really feel comfortable doing that because a larger fraction of my personal risk is in safer vehicles.
Atop that, I also think a lot about liquidity time bounds. I want access to a reasonable amount of highly-liquid, low-risk investments. I need that flexibility to be safe in the event that I need to buy something.
To my eye at least, I qualitatively differentiate between speculative investments and these liquid/low risk ones. If I felt I only had one kind of risk, I would seek out the other in some proportion.