Effect size

Pearson r or correlation coefficient

Pearson's correlation, often denoted r and introduced by Karl Pearson, is widely used as an effect size when paired quantitative data are available; for instance if one were studying the relationship between birth weight and longevity. The correlation coefficient can also be used when the data are binary. Pearson's r can vary in magnitude from −1 to 1, with −1 indicating a perfect negative linear relation, 1 indicating a perfect positive linear relation, and 0 indicating no linear relation between two variables. Cohen gives the following guidelines for the social sciences:^[10][17]

Cohen's f²

Cohen's f² is one of several effect size measures to use in the context of an F-test for ANOVA or multiple regression. Its amount of bias (overestimation of the effect size for the ANOVA) depends on the bias of its underlying measurement of variance explained (e.g., R², η², ω²).

The f² effect size measure for multiple regression is defined as: $${\displaystyle f^{2}={R^{2} \over 1-R^{2}}}$$ where R² is the squared multiple correlation.

Likewise, f² can be defined as: $${\displaystyle f^{2}={\eta ^{2} \over 1-\eta ^{2}}}$$ or $${\displaystyle f^{2}={\omega ^{2} \over 1-\omega ^{2}}}$$ for models described by those effect size measures.^[20]

The ${\displaystyle f^{2}}$ effect size measure for sequential multiple regression and also common for PLS modeling^[21] is defined as: $${\displaystyle f^{2}={R_{AB}^{2}-R_{A}^{2} \over 1-R_{AB}^{2}}}$$ where R²_A is the variance accounted for by a set of one or more independent variables A, and R²_AB is the combined variance accounted for by A and another set of one or more independent variables of interest B. By convention, f² effect sizes of ${\displaystyle 0.1^{2}}$, ${\displaystyle 0.25^{2}}$, and ${\displaystyle 0.4^{2}}$ are termed small, medium, and large, respectively.^[10]

Cohen's ${\displaystyle {\hat {f}}}$ can also be found for factorial analysis of variance (ANOVA) working backwards, using: $${\displaystyle {\hat {f}}_{\text{effect}}={\sqrt {(F_{\text{effect}}df_{\text{effect}}/N)}}.}$$

In a balanced design (equivalent sample sizes across groups) of ANOVA, the corresponding population parameter of ${\displaystyle f^{2}}$ is $${\displaystyle {SS(\mu _{1},\mu _{2},\dots ,\mu _{K})} \over {K\times \sigma ^{2}},}$$ wherein μ_j denotes the population mean within the j^th group of the total K groups, and σ the equivalent population standard deviations within each groups. SS is the sum of squares in ANOVA.

Cohen's q

Another measure that is used with correlation differences is Cohen's q. This is the difference between two Fisher transformed Pearson regression coefficients. In symbols this is $${\displaystyle q={\frac {1}{2}}\log {\frac {1+r_{1}}{1-r_{1}}}-{\frac {1}{2}}\log {\frac {1+r_{2}}{1-r_{2}}}}$$

where r₁ and r₂ are the regressions being compared. The expected value of q is zero and its variance is $${\displaystyle \operatorname {var} (q)={\frac {1}{N_{1}-3}}+{\frac {1}{N_{2}-3}}}$$ where N₁ and N₂ are the number of data points in the first and second regression respectively.

Standardized mean difference

A (population) effect size θ based on means usually considers the standardized mean difference (SMD) between two populations^[22]: 78 $${\displaystyle \theta ={\frac {\mu _{1}-\mu _{2}}{\sigma }},}$$ where μ₁ is the mean for one population, μ₂ is the mean for the other population, and σ is a standard deviation based on either or both populations.

In the practical setting the population values are typically not known and must be estimated from sample statistics. The several versions of effect sizes based on means differ with respect to which statistics are used.

This form for the effect size resembles the computation for a t-test statistic, with the critical difference that the t-test statistic includes a factor of ${\displaystyle {\sqrt {n}}}$. This means that for a given effect size, the significance level increases with the sample size. Unlike the t-test statistic, the effect size aims to estimate a population parameter and is not affected by the sample size.

SMD values of 0.2 to 0.5 are considered small, 0.5 to 0.8 are considered medium, and greater than 0.8 are considered large.^[23]

Cohen's d

Cohen's d is defined as the difference between two means divided by a standard deviation for the data, i.e. $${\displaystyle d={\frac {{\bar {x}}_{1}-{\bar {x}}_{2}}{s}}.}$$

Jacob Cohen defined s, the pooled standard deviation, as (for two independent samples):^[10]: 67 $${\displaystyle s={\sqrt {\frac {(n_{1}-1)s_{1}^{2}+(n_{2}-1)s_{2}^{2}}{n_{1}+n_{2}-2}}}}$$ where the variance for one of the groups is defined as $${\displaystyle s_{1}^{2}={\frac {1}{n_{1}-1}}\sum _{i=1}^{n_{1}}(x_{1,i}-{\bar {x}}_{1})^{2},}$$ and similarly for the other group.

The table below contains descriptors for magnitudes of d = 0.01 to 2.0, as initially suggested by Cohen (who warned against the values becoming de facto standards, urging flexibility of interpretation) and expanded by Sawilowsky.^[11]

Other authors choose a slightly different computation of the standard deviation when referring to "Cohen's d" where the denominator is without "-2"^{[24][25]: 14} $${\displaystyle s={\sqrt {\frac {(n_{1}-1)s_{1}^{2}+(n_{2}-1)s_{2}^{2}}{n_{1}+n_{2}}}}}$$ This definition of "Cohen's d" is termed the maximum likelihood estimator by Hedges and Olkin,^[22] and it is related to Hedges' g by a scaling factor (see below).

With two paired samples, we look at the distribution of the difference scores. In that case, s is the standard deviation of this distribution of difference scores. This creates the following relationship between the t-statistic to test for a difference in the means of the two groups and Cohen's d: $${\displaystyle t={\frac {{\bar {X}}_{1}-{\bar {X}}_{2}}{\text{SE}}}={\frac {{\bar {X}}_{1}-{\bar {X}}_{2}}{\frac {\text{SD}}{\sqrt {N}}}}={\frac {{\sqrt {N}}({\bar {X}}_{1}-{\bar {X}}_{2})}{SD}}}$$ and $${\displaystyle d={\frac {{\bar {X}}_{1}-{\bar {X}}_{2}}{\text{SD}}}={\frac {t}{\sqrt {N}}}}$$

Cohen's d is frequently used in estimating sample sizes for statistical testing. A lower Cohen's d indicates the necessity of larger sample sizes, and vice versa, as can subsequently be determined together with the additional parameters of desired significance level and statistical power.^[26]

For paired samples Cohen suggests that the d calculated is actually a d', which does not provide the correct answer to obtain the power of the test, and that before looking the values up in the tables provided, it should be corrected for r as in the following formula:^[27] $${\displaystyle d={\frac {d'}{\sqrt {1-r}}}.}$$

Glass' Δ

In 1976, Gene V. Glass proposed an estimator of the effect size that uses only the standard deviation of the second group^[22]: 78 $${\displaystyle \Delta ={\frac {{\bar {x}}_{1}-{\bar {x}}_{2}}{s_{2}}}}$$

The second group may be regarded as a control group, and Glass argued that if several treatments were compared to the control group it would be better to use just the standard deviation computed from the control group, so that effect sizes would not differ under equal means and different variances.

Under a correct assumption of equal population variances a pooled estimate for σ is more precise.

Hedges' g

Hedges' g, suggested by Larry Hedges in 1981,^[28] is like the other measures based on a standardized difference^[22]: 79 $${\displaystyle g={\frac {{\bar {x}}_{1}-{\bar {x}}_{2}}{s^{*}}}}$$ where the pooled standard deviation ${\displaystyle s^{*}}$ is computed as: $${\displaystyle s^{*}={\sqrt {\frac {(n_{1}-1)s_{1}^{2}+(n_{2}-1)s_{2}^{2}}{n_{1}+n_{2}-2}}}.}$$

However, as an estimator for the population effect size θ it is biased. Nevertheless, this bias can be approximately corrected through multiplication by a factor $${\displaystyle g^{*}=J(n_{1}+n_{2}-2)\,\,g\,\approx \,\left(1-{\frac {3}{4(n_{1}+n_{2})-9}}\right)\,\,g}$$ Hedges and Olkin refer to this less-biased estimator ${\displaystyle g^{*}}$ as d,^[22] but it is not the same as Cohen's d. The exact form for the correction factor J() involves the gamma function^[22]: 104 $${\displaystyle J(a)={\frac {\Gamma (a/2)}{{\sqrt {a/2\,}}\,\Gamma ((a-1)/2)}}.}$$ There are also multilevel variants of Hedges' g, e.g., for use in cluster randomised controlled trials (CRTs).^[29] CRTs involve randomising clusters, such as schools or classrooms, to different conditions and are frequently used in education research.

Ψ, root-mean-square standardized effect

A similar effect size estimator for multiple comparisons (e.g., ANOVA) is the Ψ root-mean-square standardized effect:^[20] $${\displaystyle \Psi ={\sqrt {{\frac {1}{k-1}}\cdot \sum _{j=1}^{k}\left({\frac {\mu _{j}-\mu }{\sigma }}\right)^{2}}}}$$ where k is the number of groups in the comparisons.

This essentially presents the omnibus difference of the entire model adjusted by the root mean square, analogous to d or g.

In addition, a generalization for multi-factorial designs has been provided.^[20]

Distribution of effect sizes based on means

Provided that the data is Gaussian distributed a scaled Hedges' g, ${\textstyle {\sqrt {n_{1}n_{2}/(n_{1}+n_{2})}}\,g}$, follows a noncentral t-distribution with the noncentrality parameter ${\textstyle {\sqrt {n_{1}n_{2}/(n_{1}+n_{2})}}\theta }$ and (n₁ + n₂ − 2) degrees of freedom. Likewise, the scaled Glass' Δ is distributed with n₂ − 1 degrees of freedom.

From the distribution it is possible to compute the expectation and variance of the effect sizes.

In some cases large sample approximations for the variance are used. One suggestion for the variance of Hedges' unbiased estimator is^{[22] : 86} $${\displaystyle {\hat {\sigma }}^{2}(g^{*})={\frac {n_{1}+n_{2}}{n_{1}n_{2}}}+{\frac {(g^{*})^{2}}{2(n_{1}+n_{2})}}.}$$

Strictly standardized mean difference (SSMD)

As a statistical parameter, SSMD (denoted as ${\displaystyle \beta }$) is defined as the ratio of mean to standard deviation of the difference of two random values respectively from two groups. Assume that one group with random values has mean ${\displaystyle \mu _{1}}$ and variance ${\displaystyle \sigma _{1}^{2}}$ and another group has mean ${\displaystyle \mu _{2}}$ and variance ${\displaystyle \sigma _{2}^{2}}$. The covariance between the two groups is ${\displaystyle \sigma _{12}.}$ Then, the SSMD for the comparison of these two groups is defined as^[30]

${\displaystyle \beta ={\frac {\mu _{1}-\mu _{2}}{\sqrt {\sigma _{1}^{2}+\sigma _{2}^{2}-2\sigma _{12}}}}.}$

If the two groups are independent,

${\displaystyle \beta ={\frac {\mu _{1}-\mu _{2}}{\sqrt {\sigma _{1}^{2}+\sigma _{2}^{2}}}}.}$

If the two independent groups have equal variances ${\displaystyle \sigma ^{2}}$,

${\displaystyle \beta ={\frac {\mu _{1}-\mu _{2}}{{\sqrt {2}}\sigma }}.}$

Other metrics

Mahalanobis distance (D) is a multivariate generalization of Cohen's d, which takes into account the relationships between the variables.^[31]

Cohen's omega (ω)

Another measure of effect size used for chi-squared tests is Cohen's omega (${\displaystyle \omega }$). This is defined as $${\displaystyle \omega ={\sqrt {\sum _{i=1}^{m}{\frac {(p_{1i}-p_{0i})^{2}}{p_{0i}}}}}}$$ where p_0i is the proportion of the i^th cell under H₀, p_1i is the proportion of the i^th cell under H₁ and m is the number of cells.

In Statistical Power Analysis for the Behavioral Sciences (1988, pp. 224–225), Cohen gives the following general guideline for interpreting omega (see table below), but warns against its "possible inaptness in any given substantive context" and advises to use context-relevant judgment instead.

More information ...

Effect Size	${\displaystyle \omega }$
Small	0.10
Medium	0.30
Large	0.50

Close

Odds ratio

The odds ratio (OR) is another useful effect size. It is appropriate when the research question focuses on the degree of association between two binary variables. For example, consider a study of spelling ability. In a control group, two students pass the class for every one who fails, so the odds of passing are two to one (or 2/1 = 2). In the treatment group, six students pass for every one who fails, so the odds of passing are six to one (or 6/1 = 6). The effect size can be computed by noting that the odds of passing in the treatment group are three times higher than in the control group (because 6 divided by 2 is 3). Therefore, the odds ratio is 3. Odds ratio statistics are on a different scale than Cohen's d, so this '3' is not comparable to a Cohen's d of 3.

Relative risk

The relative risk (RR), also called risk ratio, is simply the risk (probability) of an event relative to some independent variable. This measure of effect size differs from the odds ratio in that it compares probabilities instead of odds, but asymptotically approaches the latter for small probabilities. Using the example above, the probabilities for those in the control group and treatment group passing is 2/3 (or 0.67) and 6/7 (or 0.86), respectively. The effect size can be computed the same as above, but using the probabilities instead. Therefore, the relative risk is 1.28. Since rather large probabilities of passing were used, there is a large difference between relative risk and odds ratio. Had failure (a smaller probability) been used as the event (rather than passing), the difference between the two measures of effect size would not be so great.

While both measures are useful, they have different statistical uses. In medical research, the odds ratio is commonly used for case-control studies, as odds, but not probabilities, are usually estimated.^[34] Relative risk is commonly used in randomized controlled trials and cohort studies, but relative risk contributes to overestimations of the effectiveness of interventions.^[35]

Risk difference

The risk difference (RD), sometimes called absolute risk reduction, is simply the difference in risk (probability) of an event between two groups. It is a useful measure in experimental research, since RD tells you the extent to which an experimental interventions changes the probability of an event or outcome. Using the example above, the probabilities for those in the control group and treatment group passing is 2/3 (or 0.67) and 6/7 (or 0.86), respectively, and so the RD effect size is 0.86 − 0.67 = 0.19 (or 19%). RD is the superior measure for assessing effectiveness of interventions.^[35]

Cohen's h

One measure used in power analysis when comparing two independent proportions is Cohen's h. This is defined as follows $${\displaystyle h=2(\arcsin {\sqrt {p_{1}}}-\arcsin {\sqrt {p_{2}}})}$$ where p₁ and p₂ are the proportions of the two samples being compared and arcsin is the arcsine transformation.

Probability of superiority

To more easily describe the meaning of an effect size to people outside statistics, the common language effect size, as the name implies, was designed to communicate it in plain English. It is used to describe a difference between two groups and was proposed, as well as named, by Kenneth McGraw and S. P. Wong in 1992.^[36] They used the following example (about heights of men and women): "in any random pairing of young adult males and females, the probability of the male being taller than the female is .92, or in simpler terms yet, in 92 out of 100 blind dates among young adults, the male will be taller than the female",^[36] when describing the population value of the common language effect size.

Effect size for ordinal data

Cliff's delta or ${\displaystyle d}$, originally developed by Norman Cliff for use with ordinal data,^{[37][dubious – discuss]} is a measure of how often the values in one distribution are larger than the values in a second distribution. Crucially, it does not require any assumptions about the shape or spread of the two distributions.

The sample estimate ${\displaystyle d}$ is given by: $${\displaystyle d={\frac {\sum _{i,j}[x_{i}>x_{j}]-[x_{i}<x_{j}]}{mn}}}$$x_{j}]-[x_{i} where the two distributions are of size ${\displaystyle n}$ and ${\displaystyle m}$ with items ${\displaystyle x_{i}}$ and ${\displaystyle x_{j}}$, respectively, and ${\displaystyle [\cdot ]}$ is the Iverson bracket, which is 1 when the contents are true and 0 when false.

${\displaystyle d}$ is linearly related to the Mann–Whitney U statistic; however, it captures the direction of the difference in its sign. Given the Mann–Whitney ${\displaystyle U}$, ${\displaystyle d}$ is: $${\displaystyle d={\frac {2U}{mn}}-1}$$

Cohen's g

One of simplest effect sizes for measuring how much a proportion differs from 50% is Cohen's g.^[10]: 147 It measures how much a proportion differs from 50%. For example, if 85.2% of arrests for car theft are males, then effect size of sex on arrest when measured with Cohen's g is ${\displaystyle g=0.852-0.5=0.352}$. In general:

${\displaystyle g=P-0.50{\text{ or }}0.50-P\quad ({\text{directional}}),}$

${\displaystyle g=|P-0.50|\quad ({\text{nondirectional}}).}$

Units of Cohen's g are more intuitive (proportion) than in some other effect sizes. It is sometime used in combination with Binomial test.