Not long ago, I was asked if there exists any general, a priori reason to distrust any research from the social sciences (the individual asking meant “social sciences” to include the behavioral/psychological sciences). Unfortunately and surprisingly, the answer is yes. Worse still, the first reason basically defines psychiatry and exists in the neurosciences and elsewhere. The problem (the first of two) is the measurement problem or the problem of measurements.
This is not to be confused with the measurement problem that is intimately related to the interpretation of quantum mechanics. THAT measurement problem is easier and harder as it concerns very REAL measurements of systems that we can’t relate to physical reality (i.e., relating the deterministic equations of state that govern the evolutions of states of quantum mechanical systems to their fundamentally probabilistic nature; a.k.a. the “projection postulate”, so-called “collapse” of the wave function, Born’s rule/law, Bohr’s principle of complementarity, etc.). But we need some caveats and explanations.
1) THE caveat
Clearly, I am not saying that all research from so many sciences is necessarily wrong, or even likely wrong (by “wrong”, I don’t mean “wrong” in the sense of Box’s “All models are wrong, but some are useful” I mean fatally flawed”). I mean that there are enough for there to exist a good reason to doubt the soundness of any research from these sciences. But this is still quite a claim, so let me get to the first reason.
2) The (non-quantum mechanical) measurement problem
By “measurement” I am referring to the measurement of constructs in the relevant sciences (for more detail, see esp. Is psychometrics pathological science?, Quantitative methods in psychology: inevitable and useless, and Measurement theory, psychology and the revolution that cannot happen). Essentially, from intelligence to religiosity to depression, so many of the things that are measured in social science research are things we assert to exist in ways that we assert that they exist that we measure according to the ways we’ve defined them to be. Consider something like a diagnosis of bipolar disorder. Time was that American and UK psychiatrists tended to diagnose the same presentation of symptoms with bipolar disorder on the one hand, and schizophrenia on the other. All that pretty much changed with the adoption of the biomedical model of mental health and the emergence of the DSM-III (the DSM is the equivalent of the ICD, and the DSM-III was the first edition to fully and completely adopt the biomedical model AND to assume the position of authoritative diagnostic manual in psychiatry). After 30+ years, we still have NO idea as to the underlying pathology of ANY psychiatric illness, but much evidence that there is no distinct pathology for any psychiatric illness.
More important is that psychiatric illnesses are constructs: psychiatrists determined that particular presentations of symptoms should be grouped in particular ways AS symptoms and correspond to particular diagnoses. In other words, they decided that if a person’s thoughts and behavior correspond to X set, then they have Y disease, regardless of whether or not there exists any evidence other than a decision to define a disease according to particular systems as a disease. In other sciences, this kind of phenomena is more pervasive and pernicious. Consider religiosity: until the past few hundred years, “religion” was a sociocultural practice, not a belief system as the concept “religion” is today. Yet hundreds of studies seek to measure “religiosity”. The same is true with measures of the political spectrum, measures of intelligence, measures of extroversion, etc. We define some property of personality, attitude, society, culture, etc., to exist in the way we define it to, and then use this definition (of the phenomena we defined into existence) to measure how it exists. Thus we not only assume what it is we are trying to study exists at all, but also use our assumptions to determine how it is we should study what we defined to exist in the ways we defined them to exist from the start.
Basically, we say X exists and has properties x1, x2, x3,…,xn, and then determine its nature by testing for x1, x2, x3,…,xn.
There is another fundamental problem pervasive throughout the social science and well beyond. However, it requires another post.