Statistical Reporting Practices: What the Research Shows

A Review of Published Meta-Analyses on Effect Sizes, Power, and Chi-Square Testing

Overview

This review synthesizes findings from major meta-analyses examining statistical reporting practices in psychology and related fields. The evidence reveals persistent gaps between recommended practices and actual reporting, with implications for research reproducibility and interpretation.

Statistical Power: A Persistent Problem

Jacob Cohen first documented the power problem in 1962, finding that abnormal psychology studies had only about 48% power to detect medium effects—essentially a coin flip. More than 50 years later, the situation has not meaningfully improved.

Power by Research Domain

*For medium effects. **For small effects, median N = 60.

Effect Size Reporting

The American Psychological Association has required effect size reporting since 2001, yet compliance remains inconsistent.

Improvement Over Time

Fritz, Scherndl & Kühberger (2012) documented a growth rate in effect size reporting of approximately 2% per year between 1990 and 2007. More recent data suggests continued improvement, with some journals now achieving near-complete compliance:

Chi-Square Test Assumptions

The chi-square test requires certain conditions to produce valid p-values. The most commonly cited rule comes from Cochran (1954):

When to Use Alternative Tests

Research by Camilli & Hopkins (1978) and others suggests that Yates' continuity correction may be overly conservative, and modern practice favors Fisher's exact test when assumptions are not met.

Implications for Researchers

Based on this review, we recommend:

1. Conduct a priori power analysis — Only 2.9% of studies report doing this, yet it's essential for adequate sample sizes
2. Always report effect sizes with confidence intervals — Required by APA since 2001, but still underreported
3. Check chi-square assumptions — Use Fisher's exact test when expected frequencies are low
4. Report exact p-values — Not just "p < .05" but the actual value
5. Interpret effect size magnitude — A significant p-value with negligible effect size has limited practical value

Tools for Better Practices

CrossTabs.com automatically calculates all recommended statistics:

• Effect sizes (Cramér's V, phi, odds ratio) with 95% confidence intervals
• Power analysis for sample size planning
• Automatic Fisher's exact test when assumptions are violated
• Expected frequency warnings
• APA-format output

References

1. Button KS, Ioannidis JPA, Mokrysz C, et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365-376. doi:10.1038/nrn3475
2. Cochran WG (1954). Some methods for strengthening the common χ² tests. Biometrics, 10(4), 417-451.
3. Cohen J (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65(3), 145-153.
4. Fritz A, Scherndl T, Kühberger A (2012). A comprehensive review of reporting practices in psychological journals. Theory & Psychology, 23, 98-122. doi:10.1177/0959354312436870
5. Mone MA, Mueller GC, Mauland W (1996). The perceptions and usage of statistical power in applied psychology and management research. Personnel Psychology, 49(1), 103-120.
6. Stanley TD, Carter EC, Doucouliagos H (2018). What meta-analyses reveal about the replicability of psychological research. Psychological Bulletin, 144(12), 1325-1346. doi:10.1037/bul0000169
7. Sun S, Pan W, Wang LL (2010). A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology. Journal of Educational Psychology, 102(4), 989-1004.
8. Szucs D, Ioannidis JPA (2017). Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS Biology, 15(3), e2000797. doi:10.1371/journal.pbio.2000797

About this review: This page synthesizes findings from peer-reviewed meta-analyses and systematic reviews. All statistics are sourced from published research. CrossTabs.com provides free statistical tools designed to support best practices in categorical data analysis.