It’s poorly understood by many who use the DSM, and without understanding how arbitrary and or subjective it can be it may be difficult to avoid “overfitting” in the clinical setting.
I'm not sure I'm particularly convinced that this is an issue with the method of factor analysis and by extension psychometrics, per-se. Unless one specifies a causal model and actually tries to do a risky test of their theory, any other method is liable to the issue of arbitrariness and subjectivity. Psychometrics itself has come a long way and there have been many advancements to put it on firmer footing. If anything, the issue isn't with the method, but by the user of the method. I don't know if I agree that it's an issue of understanding a method, rather than an over-reliance on data (analysis) over theoretical guidance and trying to take a hammer to theories.
It’s not a problem with FA, it’s a problem with people using the DSM who don’t understand how the math behind it influences what they are doing. Ditto for IQ. If you use IQ measures professionally, you should grok FA.
But what would you have them do instead of FA? I think we're partially agreeing here, but my thinking is that no analytical technique on its-own will be a panacea whether the users really understand it or not. Why would increasing their understanding of the technique affect what they do, when there's not really any other truly different methodological alternative?
