It's parallax, basically. The pigment dots and mask holes are positioned such that when you look from the perspective of the "red" electron gun (*), you only see red pigment dots. Move a couple cm to the "blue" gun and the parallax shift now makes you to see only blue pigment dots instead. Or from the other direction, no matter which "red" dot you stand at, you only see the "red" gun through "your" hole.
The exact sizes, shapes, and positions of the pigment dot triples (and/or the mask holes) are presumably chosen so that this holds even away from the main axis. Also, the shape of the deflecting field is probably tuned to keep the rays as well-focused as possible. Similarly to how photographic lenses are carefully designed to minimize aberrations and softness even far from the optical axis.
(*) Simplifying a bit by assuming that the beam gets deflected immediately as it leaves the gun, which is of course inaccurate.