At the same time they limited it to a harvard calculus based physics class using the "regular instructors approach used for years" so one professor teaching calc based physics at harvard. Its making an assumption that this would work for every other academic setting and that there isn't some bias inherent with either the teacher, the coursework, or even the student body of harvard which is arguably more selective than a lot of other student bodies. Does this even hold water for other physics classes at harvard? I seem to remember from my own schooling quite a lot of variance between different professors exam scores on the same exact course in the same exact semester, something rectified with a huge curve at the end of the semester. In other words, maybe this professor is just a terrible lecturer. We don't have very strong evidence to say one way or another about the applicability of this study to the general population of students, I don't think.

Correction: Two professors teaching non-honors mechanics at Harvard. Authors' claim that using the non-honors course filtered out the "most well-prepared" students. The class appears to count as credit towards a physics degree and 1/3 of the class went on to pursue a physics degree.

Sure it's calculus-based, but non-calculus based mechanics is, IMO, educational malpractice at the collegiate level. You can do a qualitative approach to mechanics without calculus, but that ought not fill up an entire semester; a "survey of physics" class could take such an approach, but a purely mechanics class ought not.

[edit]

Also, the marginal student at Harvard is going to be quite similar to a student at any number of only-slightly-less selective schools. I could see an argument that this might not replicate at a large non-selective state school, but (assuming it replicates at Harvard), I'd assume it would replicate across a wide swath of selective schools.