>they can't otherwise punish you for refusing to give them your password.

Except of course by denying you entry, marking you in some "no fly" blacklist, and other ways that are not oficially "punishments", but are very much so in practice...