It just Recommends:libpam-systemd, its not a strict dependency. If I try to do an 'apt-get remove systemd systemd-shim' it doesn't show that it would remove XFCE. I'm on KDE now though so I haven't checked how well xfce works without systemd. KDE itself works quite well with systemd-shim.