Well libstd has unsafe all over the place. And as OP points out it catches this unsoundness issue in the language even though there's no unsafe anywhere, so it goes beyond just "dip into unsafe".
Sure, but the unsafe in libstd is some of the most tested unsafe in existence. And sure, there are some soundness bugs, but in my experience they are edge cases that rarely come up. I don't think I have personally ever run into one in the wild.
Regardless, it is not feasible for you to run your code under miri all the time, so you have to pick and choose where you use it, as a practical matter.