How to secure your new VPS: a step-by-step guide

You should always verify that SSH password auth is actually off; run

  ssh -v myserver : 2>&1 | grep continue

and ensure that it only gives "publickey"!

(A surprising number of VPSs will re-enable passwords in a .d config file. And really, even if you've checked for that, the extra 10 seconds to make sure is worth it.)

Good advice. Another option:

  ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no myserver