I use it to inspect video frames by frames, particularly being able to go back one frame. VLC doesn't support it, this thread about the feature is hilarious https://forum.videolan.org/viewtopic.php?t=120627
To do this well you need to keep the old frames around...or go back to the previous keyframe and re-render. That might be hard if your design is playback-optimized.
Again: There is no reason to start caching previous frames until AFTER the user has paused and pressed the "back frame" key. Only THEN does it need to rewind to the previous i-frame and re-render and cache frames. And there is no measurable cost to remembering the timestamp of the last i-frame, so you know where to rewind to.