I bet you're using Chrome. Open a video in Chrome and the video is immediately playable, load the same video on the same machine in Firefox and you can expect to wait 5+ seconds for the video to be playable.
I suspect that non-Chrome browsers are being intentionally hobbled
I did a pretty low rigor test and just pulled up one of those 4K videos with the swirling ink in Firefox on my M1 Mac and it seemed to load just as fast. The only difference was that it didn't autoplay the video (because I'm logged out I think) but I clicked it as the page loaded and it played instantly.
I don't doubt at all that Google hobbles their sites on Firefox but at least on my machine they aren't doing a great job of it
I suspect that non-Chrome browsers are being intentionally hobbled