Exactly, then you can broaden the scope of "harm" to be whatever is politically convenient. For example, I could turn your same argument around and use it to suppress any discussion of abortion whatsoever because the life of the embryo has precedence over any speech that would jeopardize it

Not that I personally think that's the right thing, just there's a danger in making core principles this ambiguous