There’s an attack vector in there somewhere.
replies(3):
2) copy unencrypted SSH host key from it to a new computer (which necessarily must not be stored in the data volume), configured with the network identity of original computer
3) leave new computer in place of original to capture remote SSH-to-unlock attempt
4) use knowledge of password to unlock original's filevault at your leisure somewhere offsite