In in the end I just went the whole hog and set up a PiKVM, so now if I mess up the machine's networking (or even completely break the OS) I can still recover it remotely even though it doesn't have a proper BMC or anything like that.
In general this approach seems ugly in principle but I really like it in practice. It lets you retrofit solid remote capabilities onto consumer hardware. That way you have such a broader market to buy from.
It's pretty funny that you have both the input and output physically installed in and powered by the chassis, but then you run cables to connect them!
It would seem reasonable to at least have the option to have it directly enumerate as a USB hub/display device on the PCIe bus it gets power from! But maybe that would add a lot of bulk/cost?