Originally, I was going to go over a few studies that I was made aware of because my sister (who graduated from the University of Pennsylvania’s School of Nursing and is now a graduate nursing student) asked for my input on studies she read for courses she’s taking to become a nurse practitioner. I still intend to, but I think I need to introduce some notions/concepts/issues first. I also need to address a deficiency in my blog posts I didn’t realize until recently. I’ve spent a lot of time covering general issues and covered specific (and mostly statistical) errors in specific studies, but haven’t really addressed how scientists go about designing a study such that they think that the data they gather can inform them as to the research question. Put simply: how an experiment tests what it is supposed to.
While the problems that occur in so many sciences that result from an inadequate grasp of data analysis and statistical methods are severe, they’re also rather technical and don’t inform the average person as to how scientists do what they do. Also, the application of inferior or problematic statistical methods can still yield meaningful, useful results, while the design of an experiment can ensure that whatever analyses are performed on the gathered data the results will be meaningless. In short, if your experiment gives you data that doesn’t actually relate to what you claim it does (for example, you give a bunch of people a questionnaire you believe measures values universal to humans like one of the studies my sister shared with me, and that questionnaire doesn’t actually measure what you think, you can use the most sophisticated mathematical methods you desire and all you are doing is manipulating meaningless values).
This post is just to prepare for any future ones that focus on experimental design, as I feel I’ve failed to emphasize how important that is. Even when criticizing the standard statistical methods that fall under “null hypothesis significance testing” (NHST) or just “hypothesis testing”, I have not adequately shown the ways in which the experimental designs are determined by the belief that such methods are required. The sciences concern empirical study, and data analysis (while essential) is really secondary. So I plan to give some examples of the nuances of experimental designs. Here, I just want to note that this is a nuanced issue, it is at the heart of the scientific endeavor, and it is nothing like the “science experiments” one encounters in high school science classes. You can’t understand scientific research without an appreciation of how scientists actually go about testing research questions, and this is nothing like most people believe.