With AM, anything that causes a variation in the intensity of the signal will introduce noise.
With FM, anything that causes a variation in the timing of the signal will introduce noise.
Unless you’re traveling at relativistic speeds, operating a time dilation device, or colocated with a black hole, you usually aren’t going to see the rate that time flows at vary.
Thus if you can make the amplitude of your signal irrelevant past a certain threshold and embed all the information into the time domain, the only thing introducing interference should be other EM sources that happen to be on the same channel.