Exploring Statistical Significance and Pitfalls in Hypothesis Testing

This essay explores the use of p < .05 in determining statistical significance, discussing its prevalence and limitations. It examines arguments for and against its use, referencing alternative significance levels and the default settings in statistical software. The essay also identifies pitfalls in statistical significance testing, including uncorrected multiple testing, publication bias, and semantic misinterpretations. Furthermore, it addresses why a p-value of 0.05 is often considered a logical trade-off in medical fields, balancing the risk of Type I errors with the need for sufficient data. The document is available on Desklib, a platform offering a variety of study tools and resources for students.

Running head: PSYCHOLOGY
Psychology
Name of the Student:
Name of the University:
Author Note:

1PSYCHOLOGY
Why do you think p < .05 is used to determine statistical significance?
It has been argued by Sullivan and Feinn (2012), that 0.05 is not the only single
significance level used for the study purpose, the other smaller values used also include 0.1 and
0.01. While it is important to mention that the p<0.05 is sometimes viewed as the hard and fast
rule or a threshold. The other significance levels of 0.1, 0.01 are used in other fields which
includes ecological literature that are often plagued by small sample sizes. Manufacturing and
engineering uses the larger sample and at the same time it is easier to obtain as well and it uses
0.01. However, many people use the 0.05 significance level because majority of the statistical
software uses 0.05 as the default value. Pearson in one of his paper gave his remarks on the p-
values. It says that the if p-value is 0.55 then it can be considered to be remarkably good; if p-
value is 0.28 then the result is fairly represented; if the p-value is 0.01 then the results are not
probable and it is not compatible with the normal sampling; and if the p-value is 0.01 then the
result is very improbable (Masicampo&Lalande, 2012).
What are the pitfalls of statistical significance or hypothesis testing?
It has been stated by Hirschauer et al. (2018), the pitfalls in statistical significance testing
includes uncorrected multiple testing which is equivalent to having statistical significance with
inflated claims. This also includes the following the covert testing methodologies of the
analytical methodologies (Sedgwick, 2014), disregard to the multiple testing; Exaggerated
emphasis on single studies and it does not have the meta- analysis along with improper meta-
analysis, and it also lack the Bayesian analysis; semantic misinterpretations which includes the
inverse probability errors (interpreting the p-value to be having the probability of a null value),
false dichotomy (interpretation of the not so significant results and the confirming the same to be

2PSYCHOLOGY
a null); publication file drawer effect and publication bias which is equivalent to the distortion
towards the positive results.
In medical fields, p < .05 is not acceptable to reject the null hypothesis. Why do you think
this is?
It has been argued by Figueiredo Filho et al. (2013), that the p-value lies within a wide
spectrum. Furthermore, it is an established fact that the smaller values of p have a tendency of
going against the null hypothesis. If the p-values are low, then there is a less likeliness that the
observe and desired results have occurred and at the same time the null hypothesis is rejected. It
is obvious that lower p-values means that there is less chance of occurrence of type I error. This
leads to a situation where the null hypothesis can be rejected falsely. The smaller values of p are
harder to achieve and it requires lots of data and due to this reason the p-value of 0.05 is used in
the medical field because it a logical trade-off.

3PSYCHOLOGY
Reference
Figueiredo Filho, D. B., Paranhos, R., Rocha, E. C. D., Batista, M., Silva Jr, J. A. D., Santos, M.
L. W. D., & Marino, J. G. (2013). When is statistical significance not
significant?.Brazilian Political Science Review, 7(1), 31-55.
Hirschauer, N., Grüner, S., Mußhoff, O., & Becker, C. (2018). Pitfalls of significance testing and
$ p $-value variability: An econometrics perspective. Statistics Surveys, 12, 136-172.
Masicampo, E. J., &Lalande, D. R. (2012). A peculiar prevalence of p values just below. 05. The
Quarterly Journal of Experimental Psychology, 65(11), 2271-2279.
Sedgwick, P. (2014). Pitfalls of statistical hypothesis testing: type I and type II errors. Bmj, 349,
g4287.
Sullivan, G. M., &Feinn, R. (2012). Using effect size—or why the P value is not enough.
Journal of graduate medical education, 4(3), 279-282.