Unless you're sure that your the "non-critical" css doesn't cause layout shifts (aka, it doesn't overload any "critical" styles), you're gonna see layout shifts even on fast connections if you load some styles at the top of the document and then do a link rel at the bottom.
I mean, i agree with you that this is insanely easy to screw up. However in most websites there is obviously css which doesn't cause reflows and is not needed for first paint. Actually separating that out correctly seems easy to mess up, but it obviously exists.