I feel like the ideal here would be for there to be a Thunderbolt/USB4 display which has multiple Thunderbolt "source" ports, and
also USB-C connectors for peripherals, where the display itself is acting as a USB controller available over Thunderbolt-PCIe, with the USB-C sockets attached to said USB controller. Change the input on the monitor, and the USB-controller PCIe card in the display would be hotplugged out of one computer and into the other.
Even more ideally, the display would also have a built-in Bluetooth controller that stays active regardless of the USB controller's attach state, such that Bluetooth peripherals could be paired to the display itself rather than to the OS (i.e. you'd manage the pairings through the OSD of the display); and then these devices would be presented through the display's USB controller as always-on direct-attached USB devices — much like VM hypervisors present host-attached Bluetooth HID devices to their VM guests. (As a side-benefit of that, as long as the computer's BIOS understood Thunderbolt well-enough to display anything during boot, then even Bluetooth peripherals would also work during boot.)