Title

Variance heterogeneity in psychological research: A Monte-Carlo study of the consequences for meta-analysis

Document Type

Poster Presentation

Publication Date

4-26-2019

Abstract

Variance heterogeneity is common in psychological research. Surveys of psychological research show that variance ratios (VRs) in two-group studies average around 2.5, with a substantial minority of studies having much higher VRs. Research has established that variance heterogeneity disturbs Type I error rates of parametric tests in primary research. Fixed-effects meta-analysis is a common statistical method in psychology for synthesizing primary research, and plays an important role in cumulative science and evidence-based practice. Little is known about the consequences of variance heterogeneity for meta-analytic estimates. The present research reports a Monte Carlo study in which the results of k = 8 or 20 primary studies were generated from each of the distributions N(100, 15) and N(106, 15), for δ = 0.40 (effect size). Variance heterogeneity was created by contaminating the second distribution with elements from a N(106, 45) distribution in proportions ranging from 0.00 to 0.25, to achieve VRs ranging from 1.0 to 3.0. Each simulated fixed-effects meta-analysis (5000 replications) yielded the following estimates: Hedges’ g, CI95% coverage, and I2. In the baseline (VR = 1.0) simulation, g = 0.40 and CI95% coverage = 0.950. In general, larger VRs at the primary-study level were associated with smaller Hedges’ gs and poorer CI95% coverage at the meta-analytic level. For example, at VR = 2.6, g = 0.30 and CI95% coverage = 0.801. In other words, a meta-analysis of studies that simulated the average VR in psychological research substantially underestimated the true effect and inflated the Type I error rate. Study-level variance heterogeneity also inflated estimates of between-study variance (I2), which has implications for meta-regression modeling. This study demonstrates that widely used meta-analytic methods do not produce accurate parameter estimates in the presence of study-level variance heterogeneity.

Comments

Presented at the Eighth Annual Conference of the Upstate Chapters of the American Statistical Association in Rochester, New York, April 26, 2019.

This document is currently not available here.

Additional Files

COinS