Comment on “One Cheer for Null Hypothesis Significance Testing”

After expressing his bafflement at the “surprisingly adamant…opposition” by many to null hypothesis significance testing (NHST), Wainer attempts to demonstrate the efficacy of NHST when used correctly. In “One Cheer for Null Hypothesis Significance Testing” (Psychological Methods), Wainer’s argument is based on the use of scientific fields and important questions within them. He provides a sample of 6 different fields and specific breakthroughs or research questions within them and claims these are not only appropriate questions for NHST, but that NHST could have provided the answers we have on these questions today. In particular, he claims that the use of NHST in e.g., physics could answer questions like the constancy of the speed of light and therefore provide the foundations for developing special relativity. He even asserts that using NHST to answer questions like those in his examples would mean studies for which some researcher or researchers  “might have been rewarded with a Nobel prize or even canonization.”

As Wainer doesn’t attempt to address the massive literature criticizing NHST, I won’t attempt to review it. Rather, I am interested in whether or to what extent Wainer is justified in claiming that NHST could have led researchers to the answers provided in Wainer’s paper. Wainer begins his sample of 6 fields and their associated specific scientific questions with physics and the speed of light. The reasoning here seems straightforward: the speed of light is equal to c regardless of reference frame and we can easily (it would seem) formulate a suitable null and alternative hypothesis. Wainer provides us with the formal statements of the null and alternative hypotheses here:

H0: ci = cj for all i and j

H1: ci cj

where “ci is the speed of light in reference frame i.” Wainer concludes: “Note that if, after credible effort when reference frames i and j are moving away from each other at great speeds, we were still unable to reject the hypothesis, we would have gone a long way toward providing the basis for the theory of relativity. Einstein would have been pleased.”

There are numerous issues at virtually every level of Wainer’s example. Many of the issues relate to the flawed, implicit assertions that 1) NHST allows researchers to determine whether the null hypothesis or the alternative is true and 2) even granting this incorrect understanding of NHST’s logic, that demonstrating the speed of light to be constant regardless of (inertial) reference frames was important. The latter problem is multilayered and somewhat nuanced.

Both special relativity and the constancy of the speed of light were arguably contained in or entailed by Galilean relativity (Petkov, 2009, Sect. 3.2-3.3). Looking closely at what could have been concluded simply using Galileo’s principle of relativity, we find that Galileo’s arguments (and empirical investigations) present a powerful argument against the absence of any absolute uniform motion. The constancy of the speed of light follows necessarily from this conclusion (were light not constant for all inertial reference frames, then it would be possible to demonstrate absolute uniform motion). Second, one possible reason for the long delay from Galilean relativity to special relativity is precisely due to something akin to NHST: “Galileo succeeded in demonstrating that any experiments of the type he discussed designed to show that the Earth is not moving would always produce null results. In other words, no matter what kind of mechanical experiments we perform, we cannot detect the uniform motion of a body in space. Let me specifically emphasize that Galileo did not disprove the Ptolemaic system of the world; what he disproved were the arguments against the Copernican system…Galileo’s principle of relativity simply states the null results of the experiments intended to detect the uniform motion of an object, but does not answer the fundamental question of why we cannot discover that motion” (Petkov, 2009, pp. 30-31).

As Petkov demonstrates, asking this question leads to serious, far-reaching challenges beyond Galileo’s devastating critique of Aristotelian mechanics. As soon as we start answering why we can’t detect the motion of the Earth using the kinds of experiments Galileo considers, we are led to ask, if the Earth is indeed moving, what is it moving in? The assumption that everything moves in a medium was part of Aristotelian mechanics. Couching Galileo’s work in the language of NHST, we might say that he assumed the null hypothesis (Aristotle’s theory of motion is true) and showed that under this assumption, we should find things to be true that we don’t (e.g., a rock dropped from the mast of a moving ship should not appear to fall straight down, because in Aristotle’s theory the of motion we should in fact be able to determine whether or not a ship is moving by whether or not the rock falls straight down as it does when we are standing “unmoving” on the ground or whether it falls away from the point straight below due to the motion of the ship). However, because he sought merely to show that an alternative hypothesis must be correct (more precisely, he sought to disprove Aristotle), he provided no evidence for a particular alternative hypothesis including the important ones required to understand just what his argument entailed.

By allowing for the Earth to be in motion, and more importantly that this is impossible to detect, Galileo opened the door to the question of “space” as a medium. A primary motivation for Aristotle’s theory of motion was to explain how e.g., a spear or arrow could continue to move in space without a “mover”. Thus nearly two millennia before Galileo the problem with space as a medium was already known. Galileo’s crushing response didn’t solve this problem. It merely rendered the previous solution untenable. He did, though, make possible the conclusion that space is a medium (by showing that an object can continue to move through space without a mover). To explain how space works as a medium compared to media such as water, metal, sand, etc., requires understanding the nature of space itself. In particular, explanation requires an ontological interpretation of space: either there is one Space in which everything moves (an absolute Space) but which we can’t detect it, or there is no such absolute Space and therefore no absolute uniform motion (for there would only exist space that things move relative or with respect to). Petkov (2009, Sect. 3.1) explores how “between the time of Galileo and twentieth century…a group of scientists as bright as Galileo” might have been able to explore these two possibilities and be led to special relativity by realizing that the second possibility is the correct one.

This didn’t happen. And there is every reason to think that no amount of NHST testing would have led scientists to conclude that the constancy of the speed of light entailed special relativity or anything else particularly interesting about mechanics or physics more generally:

1) Special relativity didn’t require the constancy of light for its development. To the extent Einstein’s 1905 paper asserted this constancy existed (see 3) below), it did so whilst acknowledging the apparent contradiction between this constancy and special relativity. Also, Einstein wasn’t the first to postulate that the speed of light was constant, a conclusion (supported by observations) that goes back to Römer and Bradley, both of whom predate Newton (Eisenstaedt, 2012).

2) Einstein himself introduced c as the constant speed of light as a postulate, and in doing so also noted that it seemed to contradict, not entail or confirm, the principle of relativity: “We will raise this conjecture (the purport of which will hereafter be called the “Principle of Relativity”) to the status of a postulate, and also introduce another postulate, which is only apparently irreconcilable with the former, namely, that light is always propagated in empty space with a definite velocity c which is independent of the state of motion of the emitting body.” (Einstein, 1905).

3) Physicists before and after Einstein initially formulated special relativity, including e.g., Lorentz (was initially given credit for special relativity alongside Einstein), continued to believe that light propagated through the “aether” or “ether”. Einstein arguably never presented a decent argument against the ether theory, and explicitly stated in later writings that his initial position on ether in 1905 “war aber zu radikal” (Einstein, 1920; cf. Einstein 1919, Kostro, 2000). What made Einstein’s formulation radical (even for Einstein, some 15 years later, who called it “too radical”) was not that the speed of light was constant, but that it propagates at a constant velocity without a medium (“in empty space”). So “successful” did physicists regard the ether theory that, were it not for the famous experiments by Michelson and Morley as well as others it is doubtful anybody would have accepted Einstein’s second postulate (Tolman, 1910; Robertson, 1949; Prokhovnik, 1974). In fact, it is not even clear that Einstein himself intended his second postulate to be a statement about the constant velocity of light, rather than merely its constancy of independent of its emission source (Schaffner, 1974). It’s not even clear that Einstein’s special relativity superseded that of Lorentz (Prokhovnik, 1974), which didn’t require c but did use the (a)ether theory.

4) The most powerful demonstration against the (a)ether theory came not from Einstein but from empirical attempts to detect it. These empirical results, unlike Galileo’s refutation of Aristotle, were failures: Michelson and Morey used the logic of the “Luminiferous Ether” hypothesis to design their experiment, but did not observe the data expected. They didn’t, however, conclude on the basis of this and earlier failures to detect variances in the velocity of light that there was no ether. Nor did they propose what Einstein postulated about light.

4) NHST relies on conditional probability; namely, given the assumption of the null hypothesis (along with assumptions built into the use of particular statistical tests, research design, etc.) NHST allows one to calculate the probability of finding the data observed (Haller & Krauss, 2002; Gigerenzer, 2004; Krämer & Gigerenzer, 2005; Balluerka, Gómez, & Hidalgo, 2005; Newman, 2008; Fidler & Loftus, 2009; Kline, 2013). It is ludicrous to suggest that centuries of Galilean relativity would be replaced with special relativity not due to extensive philosophical, logical, and empirical results from the development of electromagnetism to superior and consistent explanations that didn’t require an increasingly problematic ether theory, but due instead to NHST. After all, at best all one could say about the Wainer’s null hypothesis (the speed of light is constant in all reference frames) is that assuming this is true, we could expect experimental tests to yield the data they would with a particular probability. Basically, because NHST can only tell you about the probability of the null hypothesis (as it requires the assumption that the null is true, and thus to the extent the null is improbable any alternative hypothesis is more probable), the conclusion that NHST would have resulted in special relativity must contend with the fact that we had a great deal of empirical evidence before Einstein, Einstein assumed but didn’t test whether his postulate was true, and nobody would have been convinced of any conclusion about either light’s velocity or the ether theory based upon some p value that assumed light has constant velocity in the first place. Einstein just assumed it without the need to conduct some pointless p value. Add to this that Galilean relativity logically contained special relativity long before Einstein and without empirical study, yet physicists were unable to formulate special relativity even WITH empirical support, and the notion that NHST could succeed where vastly superior methods & evidence failed is laughable.

Similar arguments could be marshalled, to varying degrees, against Wainer’s other examples. The real question, however, is why—given the suitability of NHST—Wainer has to rely on examples where NHST wasn’t used but perhaps could have been rather than examples of breakthroughs that actually relied on NHST.

Balluerka, N., Gómez, J., & Hidalgo, D. (2005). The controversy over null hypothesis significance testing revisited. Methodology, 1(2), 55-70.

Einstein, A. (1919). 165. To Hendrik A. Lorentz. In D. K. Buchwald, R. Schulmann, J. Illy, D. J. Kennefick, & T. Sauer (Eds.). The Collected Papers of Albert Einstein, Volume 9: The Berlin Years: Correspondence, January 1919 – April 1920 (pp. 232-234). Princeton University Press.

Einstein, A. (1920). Grundgedanken und Methoden der Relativitätstheorie in ihrer Entwicklung dargestellt. In M. Janssen, R. Schulmann, J. Illy, C. Lehner, & D. K. Buchwald (Eds.). The Collected Papers of Albert Einstein, Volume 7:The Berlin Years: Writings, 1918-1921 (pp. 245-281). Princeton University Press.

Eisenstaedt, J. (2012). The Newtonian Theory of Light Propagation. In C. Lehner, J., Renn, & M. Schemmel (Eds.). Einstein and the Changing Worldviews of Physics (Einstein Studies Vol. 12) (pp. 23-37). Birkhäuser.

Fidler, F., & Loftus, G. R. (2009). Why figures with error bars should replace p values: Some conceptual arguments and empirical demonstrations. Zeitschrift für Psychologie/Journal of Psychology, 217(1), 27-37.

Gigerenzer, G. (2004). Mindless statistics. The Journal of Socio-Economics, 33(5), 587-606.

Gigerenzer, G., Krauss, S., & Vitouch, O. (2004). The null ritual. In D. Kaplan (Ed.). (2004). The Sage handbook of quantitative methodology for the social sciences (pp. 391–408).

Haller, H., & Krauss, S. (2002). Misinterpretations of significance: A problem students share with their teachers. Methods of Psychological Research, 7(1), 1-20.

Kline, R. B. (2013). Beyond Significance Testing: Statistics Reform in the Behavioral Sciences. American Psychological Association.

Kostro, L. (2000). Einstein and the Ether. Apeiron.

Newman, M. C. (2008). “What exactly are you inferring?” A closer look at hypothesis testing. Environmental Toxicology and Chemistry, 27(5), 1013-1019.

Petkov, V. (2009). Relativity and the Nature of Spacetime (2nd Ed.). Springer.

Robertson, H. P. (1949). Postulate versus observation in the special theory of relativity. Reviews of modern Physics, 21(3), 378.

Tolman, R. C. (1910). The second postulate of relativity. Physical Review (Series I), 31(1), 26.

Schaffner, K. F. (1974). Einstein versus Lorentz: research programmes and the logic of comparative theory evaluation. British journal for the Philosophy of Science, 45-78.

Wainer, H. (1999). One cheer for null hypothesis significance testing. Psychological Methods, 4(2), 212.

This entry was posted in Physics, Statistics, The Scientific Method and tagged , , , , , , , . Bookmark the permalink.

4 Responses to Comment on “One Cheer for Null Hypothesis Significance Testing”

  1. Jeff says:

    “e.g., a rock dropped from the mast of a moving ship should not appear to fall straight down, because in Aristotle’s theory the of motion we should in fact be able to determine whether or not a ship is moving by whether or not the rock falls straight down as it does when we are standing “unmoving” on the ground or whether it falls away from the point straight below due to the motion of the ship”

    Just provokes a response and makes me think.

    So in the space-time continuum, everything we think of as moving is really an illusion, and everything that appears to be still is actually moving. Johnny and Bobby Sue are standing outside near a park bench, looking up at the clouds. Even though Johnny and Bobby Sue are standing, they are actually moving through the space-time continuum, because they have their feet planted on a giant mass called Earth. Which spins at 465.1 m/s or 1,000 miles per hour and travels around the sun at a speed of 60,000+ mph.

    So, are the clouds giving an illusion of motion as the Earth spins, rotates and makes its way around the Sun? Making Johnny and Bobby Sue buy into the notion of motion? That they are actually standing still when they are moving through space-time?

    … and of course the opposite of this is someone acknowledging they are standing still (without thinking twice about it – it’s just a given even though it’s not always a constant). Well, at least it seems that way.

    Hmm… Maybe Wainer needed a none-NHST dualist approach? His approach is equivalent to someone throwing a boomerang forwards, it going out-of-sight and coming back around to hit him in the back of the head. Or one that can make a butterfly effect, which isn’t impossible as far as I known due to natural motion without interference.

    • “So in the space-time continuum, everything we think of as moving is really an illusion, and everything that appears to be still”

      I think Auden most appropriate here:
      “But all the clocks in the city
      Began to whirr and chime
      ‘O let not time deceive you
      You cannot conquer time.”

      Time is still something of a mystery in physics, and spacetime more so. It seems clear (but isn’t exactly) that Einstein and his contemporaries (at least the ones that excepted the notion of spacetime) largely regarded it as no more “real” than any other mathematical space convenient for physics. Indeed, Einstein initially resisted the work of Minkowski (his teacher) to formulate Einstein’s algebraic “spacetime” with an actual 4D “space” (Minkowski space). The reason he yielded seems to be because he realized the great practicality of Minkowski’s geometric formulation over his own algebraic. Also, the other big contributor/founder of special relativity, Lorentz (the reason that even today in special relativity we refer constantly to “Lorentz transforms”) didn’t even buy into Einstein’s postulate about light.

      Formally, in spacetime (as formulated in special or general relativity) everything is always in motion, and motion occurs in 4D spacetime (this is less true of general relativity, or rather things are a bit more nuanced for general relativity, but it suffices).

      Currently, there are certainly those who hold that special relativity or just relativity entail that our reality is 4D (although general relativity is–general speaking– only LOCALLY Minkowskian). More simply, spacetime isn’t the 3D space we experience with an extra time dimension that makes the mathematics and geometry more precise, but rather we are all “hunks” of 4D matter. This creates a number of problems from the perspective of physics, the cognitive sciences, metaphysics, and philosophy. For example, according to this view, given the “now” of any reference frame (say, the one you have when reading this), there is some (infinite) set of other references frames for which your “now” occurs in the past. Likewise, there are reference frames for your “now” which are wayyy in the future for some other reference frames.
      This is the basis for the “block universe” interpretation of spacetime: all that objects/systems/etc. are consists of trajectories in 4D space that “slice up” the same all-encompassing spacetime “block”, and everything that has ever happened or will ever happen from a particular “perspective (“point” or “region” in this 4D space) will happen or has already happened from another.
      Perhaps the most basic, most seemingly trivial yet anything but trivial, most problematic issue with understanding reality as fundamentally and indivisibly 4D is consciousness(by indivisibly, I mean the same way that in the 3D reality we experience nothing is ACTUALLY 2D. A sheet of paper may appear to be 2D, and we might use a dry-erase board or chalkboard to draw things in “2D”, but in our 3D reality even the remnants of a pen on paper or chalk on a chalkboard exist only and always in 3D). It’s one thing to refer to reality as contrary to our experiences. For example, systems in quantum physics act in ways that have no correspondence to our experience and seem to violate basic logic. But we don’t experience the microscopic realm at all. It takes a microscope just for us to see things that are well-beyond the (albeit fuzzy) quantum-classical limit. Likewise, we don’t experience gravitation/gravitons/spacetime curvature (or whatever is responsible for the attraction between matter that we usually just call gravity), because the pull between us and the earth overwhelms the “gravitational pull” between us and trees, mountains, rocks, other people, even the moon and sun (the moon’s “gravity” influences us by e.g, tidal flow and of course the whole planet circles the sun thanks to “gravity”, but we don’t really experience this). However, time is fundamental to not only our experience, but to language and consciousness. It is difficult if not impossible to formulate a reality in which there is neither present, past, nor future in any language. It is difficult to even account for the “continuity of mind” (the experience of a present self distinguished from the past but even more importantly from the future) in a “block universe” of no continuity with respect to time.

      For time and space to be truly unified as “spacetime”, nothing in reality can be 3D, and describing e.g., my brain spatially (3-dimensionally) in this spacetime would be like describing it as a plane in 3D space. Also, the evidence for the big bang is problematic (as is expansion), because there should be some reference frame in spacetime for which the moment of the big bang is in the future or at least an equivalence between the origins of the cosmos and the expansion of the universe and a 4D block universe. Not only that, but the various physical bases for the “arrow of time” (e.g., entropy, time-irreversible systems/processes, etc.) suddenly evaporate, and usually without something to replace them (actually, in general relativity instead of time-irreversibility we find the possibility of closed timelike curves (CTCs) which permit e.g., a person to prevent themselves from being born WITHOUT using either a DeLorean or even a flux capacitor!

      Put simply, there is no “simply” when it comes to time (and by extension spacetime).

  2. Jeff says:

    I said: “which isn’t impossible as far as I known due to natural motion without interference.” Typo, meant to say possible.

  3. Jeff says:

    Although now that I’m thinking about it… hmm… makes me think more than I probably should have a few beers.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s