To me it seems like the complexity is just irreducible. There's so many formats, so many bits and pieces that can go in a video stream, they're not very visualizable, and they have surprising edge case interactions. Not to mention there's a lot of "normally the program figures this out for you, but there's an option to override it if broken" knobs and dials.
Good UX is not about reducing complexity, nor is it about hiding complexity. It's about surfacing the exact utility a user needs in the exact context they are best suited to understand each of its inputs entirely (with the least friction in generating those inputs for the user). It's very hard to do. So much so that describing 'what is wrong' with a UX would be almost as burdensome as just designing a better UX. So I'm not going to tell you what is wrong with it; you KNOW what is wrong with it. It could be better. That you can't specify as to how just means that you aren't currently undertaking the complicated process of redesigning it. It doesn't mean you don't know good UX from bad UX.
Now, all of that aside, I do like Handbrake and I do think it offers a ton of functionality with so little friction that it's one of my very favorite and most-used apps. No login, no project setup, no x, no y, no z. Just a thin wrapper around a badass command line utility, with tons of options for users to override, and sensible defaults. There's a lot to love about Handbrake!
But "my grandma can use it", or "a plumber can use it", or "a person who doesn't understand the technicals and just needs to do one stupid thing that the app can definitely do, can use it" are signs of good UX. You wouldn't say any of those things about Handbrake.
In my experience, handbrake doesn't expose every option from ffmpeg and is more focused on transcoding. One nitpick I have with handbrake is that it doesn't support VAAPI encoding nor Vulkan Video Encoding for AMD cards on Linux.
To me it seems like the complexity is just irreducible. There's so many formats, so many bits and pieces that can go in a video stream, they're not very visualizable, and they have surprising edge case interactions. Not to mention there's a lot of "normally the program figures this out for you, but there's an option to override it if broken" knobs and dials.