Also consider the speed of light is also the speed of causality. If there was no such limit it means it would be possible for effects to precede causes which would lead to a very different kind of universe!
The possibility of delays being zero does not imply that negative delays are possible.
Actually, it does. Because of relativity events that occur at the same time in one frame of reference do not occur at the same time in another. A delay of zero between two different points implies that there is a reference frame where the delay is negative.
I'm not sure that holds when you take the speed of light to be infinite. Depending on which end you look at it, you'll either be dividing by zero or having infinite energy, so I don't think relativity the way we understand it would still make sense in any way.