E-mail or sms codes are not 2fa then either, if the attacker has your device (presumably with the e-mail app logged in already and the password saved). But this seems like a dubious distinction, its like saying 2fa is no longer 2fa if the attacker has access to the second factor. Thats not particularly remarkable.
You can call it 2sv, though. Two step verification. But a user can certainly chose to use in a way that makes it 2fa by storing the totp secret on a dedicated device. The bottom line for most use cases is that it stops people from getting in even if they guess or crack your password.
With hardware tokens, it still has tradeoffs. What happens when the “user” (read attacker) claims they lost or damaged the yubi key? What factor do you use to verify them before sending a new yubikey in the mail? What happens if someone breaks into the user’s mail? Etc. no method is perfect.