> So my theory is that probability is an ill-defined, unfalsifiable concept. And yet, it seems to model aspects of the world pretty well, empirically.
I have privately come to the conclusion that probability is a well-defined and testable concept only in settings where we can argue from certain exact symmetries. This is the case in coin tosses, games of chance and many problems in statistical physics. On the other hand, in real-world inference, prediction and estimation, probability is subjective and much less quantifiable than statisticians (Bayesians included) would like it to be.
> However, might it be leading us astray?
Yes, I think so. I increasingly feel that all sciences that rely on statistical hypothesis testing as their primary empirical method are basically giant heaps of garbage, and the Reproduciblity Crisis is only the tip of the iceberg. This includes economics, social psychology, large swathes of medical science, data science, etc.
> Consider the statement p(X) = 0.5 (probability of event X is 0.5). What does this actually mean? It it a proposition? If so, is it falsifiable? And how?
I'd say it is an unfalsifiable proposition in most cases. Even if you can run lots of cheap experiments, like with coin tosses, a million runs will "confirm" the calculated probability only with ~1% precision. This is just lousy by the standards of the exact sciences, and it only goes downhill if your assumptions are less solid, the sample space more complex, or reproducibility more expensive.