Nope. You own your keys, and you can switch to a different server instantly while retaining both your username and your followers. They can't impersonate you because they don't have your keys. They can't ready your messages because they don't have your keys.

It's a bit more complex underneath (e.g. the are light-clients that you can entrust with part of your key), but that's the gist.