Legacy would be pdflatex in this case, which generates PDFs directly and works with unicode without issues. However, I would also be interested in cases which xelatex doesn't handle too well.
These are purely my own opinions and not official positions of any groups or organizations I am associated with.
The biggest issues with XeTeX (compared mostly with LuaTeX, some of them also apply for pdfTeX) in my opinion are (roughly ordered for importance)
1. the missing support for microtypesetting features, especially font expansion
2. Missing support for new OpenType features. I consider especially variable fonts to be important, but I know that there are also many users missing color font features.
3. The font selection system which uses completely different lookup paths for font names and filenames leads to extreme confusion for users, especially since it's system dependent.
4. It shares the problem with older engines that many things like colors, underlines, etc. require special invisible nodes which can influence line-breaking in unexpected ways.
5. (From a programmers perspective) adding advanced PDF features under XeTeX tends to be much more complicated because of the DVI based backend which makes it harder to control the PDF file directly.
That being said, I would take XeTeX over pdfTeX any day, but I don't see much reason to use it if LuaTeX is available as an alternative.
The reason I switched from LuaTeX, which I agree is otherwise superior (and...the Lua thing!), is that it has some horrible and well-known bug in its font loading mechanism, where if you load a large number of fonts, it sucks up enormous memory and CPU resources. I have documents that simply grind to a halt if I try to process them with LuaTeX, but compile in seconds using XeLaTeX. I hope that’s fixed now, because I would rather use LuaTeX.
The issue is not so much about many fonts, but about one big font. The first time a font is loaded it has to be analyzed and cached and for some fonts this leads to excessive resource usage. You can avoid that in recent versions by using a HarfBuzz based font-shaper instead of the Lua version: Replace `\setmainfont{Some name}` with `\setmainfont[Renderer=HarfBuzz]{Some name}`.
The first time a bug font is used you might still see a slight delay (~1 second?), but it's orders of magnitude faster and does not require so much resources.
