You need recent ssh-keygen's -o to make the passphprase effective (default for ed25519). That was covered here at some stage, but I don't have a reference.