I recommend using mpv with the open-with extension instead (right click -> open with -> mpv.) Thanks to youtube-dl, mpv will handle most any video website except netflix. If you tell mpv to stay-on-top, the effect essentially picture-in-picture, but with the full capability of mpv. You can drag new links into mpv to change what video is playing (hold down shift while dragging in a new link to append to the playlist.)
I tried this because FF doesn't do hardware video decoding yet, and, on an older laptop, FF was taking up a whole core for watching YouTube vs. around 30% in mpv.
Unfortunately every "open with.." solution I tried though took 10-15 seconds to open the video in mpv, which turned out to be even more frustrating than the problem.
I've experienced what you describe. I think the problem is most severe when you're on a high-latency connection, such as a VPN to an endpoint far away, because there are a few round trips in the process with youtube-dl. I don't really have a solution to that, but using --force-window=immediate can help a little (it will make a blank mpv window appear immediately, before the video is ready, so at least you aren't left wondering whether you misclicked prior to the video starting.)
https://github.com/darktrojan/openwith