The gold standard for this would be a Hardware Security Module (HSM), which is essentially a device that stores private keys with certain guarantees of physical security (e.g, that private key material cannot be extracted from the device once it has been generated or placed there, and the device performs operations using the key material on behalf of some client).
HSMs in various forms underpin all sorts of cryptosystems that society depends on, because securing private key material at rest is essential. You'll find them everywhere from your debit/credit card, to certificate authorities, financial institutions, defense, and your smartphone.
For your use case, I'd recommend taking a look at Yubikeys. I did a writeup a while back on how to use them to store different types of private keys for various purposes: