> But there seems to be speech that should be forbidden, like threats of physical violence.

The thing being prohibited there isn't the speech. Prohibiting threats is really prohibiting coercion. The words of the threat are the evidence of the coercion, not the crime itself.

That's why the threat has to be credible. It's not a crime to say threatening things as an actor in a play because you're not actually threatening anyone even though the communicative aspect is the same.