Also consider the speed of light is also the speed of causality. If there was no such limit it means it would be possible for effects to precede causes which would lead to a very different kind of universe!
Antimatter is often described as matter going the opposite direction in time, and this seems to hold for particle interactions and quantum physics. It breaks down for thermodynamics… but might this be because of the inherent time vector of the observer?