After expressing his bafflement at the “surprisingly adamant…opposition” by many to null hypothesis significance testing (NHST), Wainer attempts to demonstrate the efficacy of NHST when used correctly. In “One Cheer for Null Hypothesis Significance Testing” (Psychological Methods), Wainer’s argument is based on the use of scientific fields and important questions within them. He provides a sample of 6 different fields and specific breakthroughs or research questions within them and claims these are not only appropriate questions for NHST, but that NHST could have provided the answers we have on these questions today. In particular, he claims that the use of NHST in e.g., physics could answer questions like the constancy of the speed of light and therefore provide the foundations for developing special relativity. He even asserts that using NHST to answer questions like those in his examples would mean studies for which some researcher or researchers “might have been rewarded with a Nobel prize or even canonization.”
As Wainer doesn’t attempt to address the massive literature criticizing NHST, I won’t attempt to review it. Rather, I am interested in whether or to what extent Wainer is justified in claiming that NHST could have led researchers to the answers provided in Wainer’s paper. Wainer begins his sample of 6 fields and their associated specific scientific questions with physics and the speed of light. The reasoning here seems straightforward: the speed of light is equal to c regardless of reference frame and we can easily (it would seem) formulate a suitable null and alternative hypothesis. Wainer provides us with the formal statements of the null and alternative hypotheses here:
H0: ci = cj for all i and j
H1: ci ≠ cj
where “ci is the speed of light in reference frame i.” Wainer concludes: “Note that if, after credible effort when reference frames i and j are moving away from each other at great speeds, we were still unable to reject the hypothesis, we would have gone a long way toward providing the basis for the theory of relativity. Einstein would have been pleased.”
There are numerous issues at virtually every level of Wainer’s example. Many of the issues relate to the flawed, implicit assertions that 1) NHST allows researchers to determine whether the null hypothesis or the alternative is true and 2) even granting this incorrect understanding of NHST’s logic, that demonstrating the speed of light to be constant regardless of (inertial) reference frames was important. The latter problem is multilayered and somewhat nuanced.
Both special relativity and the constancy of the speed of light were arguably contained in or entailed by Galilean relativity (Petkov, 2009, Sect. 3.2-3.3). Looking closely at what could have been concluded simply using Galileo’s principle of relativity, we find that Galileo’s arguments (and empirical investigations) present a powerful argument against the absence of any absolute uniform motion. The constancy of the speed of light follows necessarily from this conclusion (were light not constant for all inertial reference frames, then it would be possible to demonstrate absolute uniform motion). Second, one possible reason for the long delay from Galilean relativity to special relativity is precisely due to something akin to NHST: “Galileo succeeded in demonstrating that any experiments of the type he discussed designed to show that the Earth is not moving would always produce null results. In other words, no matter what kind of mechanical experiments we perform, we cannot detect the uniform motion of a body in space. Let me specifically emphasize that Galileo did not disprove the Ptolemaic system of the world; what he disproved were the arguments against the Copernican system…Galileo’s principle of relativity simply states the null results of the experiments intended to detect the uniform motion of an object, but does not answer the fundamental question of why we cannot discover that motion” (Petkov, 2009, pp. 30-31).
As Petkov demonstrates, asking this question leads to serious, far-reaching challenges beyond Galileo’s devastating critique of Aristotelian mechanics. As soon as we start answering why we can’t detect the motion of the Earth using the kinds of experiments Galileo considers, we are led to ask, if the Earth is indeed moving, what is it moving in? The assumption that everything moves in a medium was part of Aristotelian mechanics. Couching Galileo’s work in the language of NHST, we might say that he assumed the null hypothesis (Aristotle’s theory of motion is true) and showed that under this assumption, we should find things to be true that we don’t (e.g., a rock dropped from the mast of a moving ship should not appear to fall straight down, because in Aristotle’s theory the of motion we should in fact be able to determine whether or not a ship is moving by whether or not the rock falls straight down as it does when we are standing “unmoving” on the ground or whether it falls away from the point straight below due to the motion of the ship). However, because he sought merely to show that an alternative hypothesis must be correct (more precisely, he sought to disprove Aristotle), he provided no evidence for a particular alternative hypothesis including the important ones required to understand just what his argument entailed.
By allowing for the Earth to be in motion, and more importantly that this is impossible to detect, Galileo opened the door to the question of “space” as a medium. A primary motivation for Aristotle’s theory of motion was to explain how e.g., a spear or arrow could continue to move in space without a “mover”. Thus nearly two millennia before Galileo the problem with space as a medium was already known. Galileo’s crushing response didn’t solve this problem. It merely rendered the previous solution untenable. He did, though, make possible the conclusion that space is a medium (by showing that an object can continue to move through space without a mover). To explain how space works as a medium compared to media such as water, metal, sand, etc., requires understanding the nature of space itself. In particular, explanation requires an ontological interpretation of space: either there is one Space in which everything moves (an absolute Space) but which we can’t detect it, or there is no such absolute Space and therefore no absolute uniform motion (for there would only exist space that things move relative or with respect to). Petkov (2009, Sect. 3.1) explores how “between the time of Galileo and twentieth century…a group of scientists as bright as Galileo” might have been able to explore these two possibilities and be led to special relativity by realizing that the second possibility is the correct one.
This didn’t happen. And there is every reason to think that no amount of NHST testing would have led scientists to conclude that the constancy of the speed of light entailed special relativity or anything else particularly interesting about mechanics or physics more generally:
1) Special relativity didn’t require the constancy of light for its development. To the extent Einstein’s 1905 paper asserted this constancy existed (see 3) below), it did so whilst acknowledging the apparent contradiction between this constancy and special relativity. Also, Einstein wasn’t the first to postulate that the speed of light was constant, a conclusion (supported by observations) that goes back to Römer and Bradley, both of whom predate Newton (Eisenstaedt, 2012).
2) Einstein himself introduced c as the constant speed of light as a postulate, and in doing so also noted that it seemed to contradict, not entail or confirm, the principle of relativity: “We will raise this conjecture (the purport of which will hereafter be called the “Principle of Relativity”) to the status of a postulate, and also introduce another postulate, which is only apparently irreconcilable with the former, namely, that light is always propagated in empty space with a definite velocity c which is independent of the state of motion of the emitting body.” (Einstein, 1905).
3) Physicists before and after Einstein initially formulated special relativity, including e.g., Lorentz (was initially given credit for special relativity alongside Einstein), continued to believe that light propagated through the “aether” or “ether”. Einstein arguably never presented a decent argument against the ether theory, and explicitly stated in later writings that his initial position on ether in 1905 “war aber zu radikal” (Einstein, 1920; cf. Einstein 1919, Kostro, 2000). What made Einstein’s formulation radical (even for Einstein, some 15 years later, who called it “too radical”) was not that the speed of light was constant, but that it propagates at a constant velocity without a medium (“in empty space”). So “successful” did physicists regard the ether theory that, were it not for the famous experiments by Michelson and Morley as well as others it is doubtful anybody would have accepted Einstein’s second postulate (Tolman, 1910; Robertson, 1949; Prokhovnik, 1974). In fact, it is not even clear that Einstein himself intended his second postulate to be a statement about the constant velocity of light, rather than merely its constancy of independent of its emission source (Schaffner, 1974). It’s not even clear that Einstein’s special relativity superseded that of Lorentz (Prokhovnik, 1974), which didn’t require c but did use the (a)ether theory.
4) The most powerful demonstration against the (a)ether theory came not from Einstein but from empirical attempts to detect it. These empirical results, unlike Galileo’s refutation of Aristotle, were failures: Michelson and Morey used the logic of the “Luminiferous Ether” hypothesis to design their experiment, but did not observe the data expected. They didn’t, however, conclude on the basis of this and earlier failures to detect variances in the velocity of light that there was no ether. Nor did they propose what Einstein postulated about light.
4) NHST relies on conditional probability; namely, given the assumption of the null hypothesis (along with assumptions built into the use of particular statistical tests, research design, etc.) NHST allows one to calculate the probability of finding the data observed (Haller & Krauss, 2002; Gigerenzer, 2004; Krämer & Gigerenzer, 2005; Balluerka, Gómez, & Hidalgo, 2005; Newman, 2008; Fidler & Loftus, 2009; Kline, 2013). It is ludicrous to suggest that centuries of Galilean relativity would be replaced with special relativity not due to extensive philosophical, logical, and empirical results from the development of electromagnetism to superior and consistent explanations that didn’t require an increasingly problematic ether theory, but due instead to NHST. After all, at best all one could say about the Wainer’s null hypothesis (the speed of light is constant in all reference frames) is that assuming this is true, we could expect experimental tests to yield the data they would with a particular probability. Basically, because NHST can only tell you about the probability of the null hypothesis (as it requires the assumption that the null is true, and thus to the extent the null is improbable any alternative hypothesis is more probable), the conclusion that NHST would have resulted in special relativity must contend with the fact that we had a great deal of empirical evidence before Einstein, Einstein assumed but didn’t test whether his postulate was true, and nobody would have been convinced of any conclusion about either light’s velocity or the ether theory based upon some p value that assumed light has constant velocity in the first place. Einstein just assumed it without the need to conduct some pointless p value. Add to this that Galilean relativity logically contained special relativity long before Einstein and without empirical study, yet physicists were unable to formulate special relativity even WITH empirical support, and the notion that NHST could succeed where vastly superior methods & evidence failed is laughable.
Similar arguments could be marshalled, to varying degrees, against Wainer’s other examples. The real question, however, is why—given the suitability of NHST—Wainer has to rely on examples where NHST wasn’t used but perhaps could have been rather than examples of breakthroughs that actually relied on NHST.
Balluerka, N., Gómez, J., & Hidalgo, D. (2005). The controversy over null hypothesis significance testing revisited. Methodology, 1(2), 55-70.
Einstein, A. (1919). 165. To Hendrik A. Lorentz. In D. K. Buchwald, R. Schulmann, J. Illy, D. J. Kennefick, & T. Sauer (Eds.). The Collected Papers of Albert Einstein, Volume 9: The Berlin Years: Correspondence, January 1919 – April 1920 (pp. 232-234). Princeton University Press.
Einstein, A. (1920). Grundgedanken und Methoden der Relativitätstheorie in ihrer Entwicklung dargestellt. In M. Janssen, R. Schulmann, J. Illy, C. Lehner, & D. K. Buchwald (Eds.). The Collected Papers of Albert Einstein, Volume 7:The Berlin Years: Writings, 1918-1921 (pp. 245-281). Princeton University Press.
Eisenstaedt, J. (2012). The Newtonian Theory of Light Propagation. In C. Lehner, J., Renn, & M. Schemmel (Eds.). Einstein and the Changing Worldviews of Physics (Einstein Studies Vol. 12) (pp. 23-37). Birkhäuser.
Fidler, F., & Loftus, G. R. (2009). Why figures with error bars should replace p values: Some conceptual arguments and empirical demonstrations. Zeitschrift für Psychologie/Journal of Psychology, 217(1), 27-37.
Gigerenzer, G. (2004). Mindless statistics. The Journal of Socio-Economics, 33(5), 587-606.
Gigerenzer, G., Krauss, S., & Vitouch, O. (2004). The null ritual. In D. Kaplan (Ed.). (2004). The Sage handbook of quantitative methodology for the social sciences (pp. 391–408).
Haller, H., & Krauss, S. (2002). Misinterpretations of significance: A problem students share with their teachers. Methods of Psychological Research, 7(1), 1-20.
Kline, R. B. (2013). Beyond Significance Testing: Statistics Reform in the Behavioral Sciences. American Psychological Association.
Kostro, L. (2000). Einstein and the Ether. Apeiron.
Newman, M. C. (2008). “What exactly are you inferring?” A closer look at hypothesis testing. Environmental Toxicology and Chemistry, 27(5), 1013-1019.
Petkov, V. (2009). Relativity and the Nature of Spacetime (2nd Ed.). Springer.
Robertson, H. P. (1949). Postulate versus observation in the special theory of relativity. Reviews of modern Physics, 21(3), 378.
Tolman, R. C. (1910). The second postulate of relativity. Physical Review (Series I), 31(1), 26.
Schaffner, K. F. (1974). Einstein versus Lorentz: research programmes and the logic of comparative theory evaluation. British journal for the Philosophy of Science, 45-78.
Wainer, H. (1999). One cheer for null hypothesis significance testing. Psychological Methods, 4(2), 212.