Science Isn’t Broken

After all, what scientists really want to know is whether their hypothesis is true, and if so, how strong the finding is. “A p-value does not give you that — it can never give you that,” said Regina Nuzzo, a statistician and journalist in Washington, D.C., who wrote about the p-value problem in Nature last year. Instead, you can think of the p-value as an index of surprise. How surprising would these results be if you assumed your hypothesis was false?

Source: Science Isn’t Broken

This article has a whole lot more on problems within science, but I just wanted to clarify that, in hypothesis testing, the p-value is not “an index of surprise” – it is your decision making tool for deciding whether a difference is there or not. You can set it at whatever level you want to argue for as an acceptable a priori chance of making a Type 1 error (a priori = decided before you do the experiment, not on the basis of your data). It has always been the case that p-values do not give you the truth of your hypothesis, but it has also always been the case that you are not actually testing the truth of your hypothesis – you are using a decision tool for establishing the “reality” of the difference between two samples by rejecting the null hypothesis (no difference). Once you have rejected the null hypothesis, you can propose an alternative hypothesis to explain the difference you have demonstrated. We accept an alternative hypothesis on the basis of the argument for it, not on the p value. The p value only gives us a basis (rejection of the null hypothesis) for considering the alternative.

In correlational data, you are looking for the degree to which you can predict one variable from another – the nature of the proposed relationship relative to the amount of variance explained is more important than its statistical significance – so much of psychology research involves p-hacking, but that is the least of the problems with so-called psychological measures based on endless likert scales.

Print Friendly, PDF & Email

Leave a Reply

Your email address will not be published. Required fields are marked *