If you don't do secure boot, you need to secure your boot chain in other ways, to prevent attacker from modifying your software to log entered passphrase.
Secure boot allows to build a verifiable chain of software (UEFI -> Bootloader -> Kernel -> Initrd) which will protect against any modification, so you can be sure that your key presses are not being logged by the malicious software. That said, commonly used Linux distros have some problems protecting initrd, but that's issue of those distros.
Another link is TPM. I set up my system in a way to keep encryption key in TPM and release it only when secure boot is enabled. This allows to decrypt root automatically, without entering passphrase and my configuration only allows to boot UKI kernel signed with my key. It trades security with convenience, of course (because now attacker, who stolen my computer, only has to break through gdm or perform other ways of attacks like extracting RAM sticks), but for me it's acceptable.