Abstract
Yuan and Chan (Psychometrika 76:670–690, 2011. doi:10.1007/S11336-011-9224-6) derived consistent confidence intervals for standardized regression coefficients under fixed and random score assumptions. Jones and Waller (Psychometrika 80:365–378, 2015. doi:10.1007/S11336-013-9380-Y) extended these developments to circumstances where data are non-normal by examining confidence intervals based on Browne’s (Br J Math Stat Psychol 37:62–83, 1984. doi:10.1111/j.2044-8317.1984.tb00789.x) asymptotic distribution-free (ADF) theory. Seven different heteroscedastic-consistent (HC) estimators were investigated in the current study as potentially better solutions for constructing confidence intervals on standardized regression coefficients under non-normality. Normal theory, ADF, and HC estimators were evaluated in a Monte Carlo simulation. Findings confirmed the superiority of the HC3 (MacKinnon and White, J Econ 35:305–325, 1985. doi:10.1016/0304-4076(85)90158-7) and HC5 (Cribari-Neto and Da Silva, Adv Stat Anal 95:129–146, 2011. doi:10.1007/s10182-010-0141-2) interval estimators over Jones and Waller’s ADF estimator under all conditions investigated, as well as over the normal theory method. The HC5 estimator was more robust in a restricted set of conditions over the HC3 estimator. Some possible extensions of HC estimators to other effect size measures are considered for future developments.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
It can be seen that the HC1-type SEs just reflect a simple denominator degrees-of-freedom adjustment to the asymptotic-based HC0 calculation that uses n in the denominator. Rationales for the less-straightforward HC2 and HC3 adjustments are clearly explained in Long and Ervin (2000) and in Hayes and Cai (2007), as well being detailed in the original publication.
The standard errors for normal theory and ADF estimators were verified against the seBeta function in Waller and Jones’ (2015) fungible package in R (R Development Core Team, 2015). The computation of \({\varvec{\Gamma }}_R \) matrices in Equation 29 used in calculating HC0, HC1, HC2, HC3, HC4, HC4M, and HC5 standard errors were verified against HC standard errors provided by the sandwich package (Zeileis, 2004) in R for unstandardized regression coefficients.
Kurtosis is expressed here by the fourth standardized cumulant \((g_4)\), which is sometimes referred to as excess kurtosis in the literature. The fourth standardized cumulant for a normal distribution equals zero. It is related to the fourth standardized moment measure of kurtosis \(B_2 =\left( {{\mu _4 }/{\sigma ^{4}}} \right) \) such that \(g_4 =B_2 -3\).
The three distributions for predictor scores cross-classified with the three kinds of error score distributions meant that excess kurtosis for the 15 million dependent variable scores ranged from 0 to 51.25. Moreover, the degree of kurtosis for \(y_i\) was amplified by increasing \(\angle {\varvec{\upbeta } }_{r^{\circ }}\) at each combination of non-normality in predictor scores and error scores.
The value of 15 million was chosen so that the population size was three times larger than the number of replications times the largest sample size used in the simulation. In contrast, Long and Erwin (2000) used 100,000 cases to define the populations in their simulations (with 1000 replications and a largest sample size of 1000).
This statement implies that robustness and accuracy are distinct but interrelated concepts (much like reliability and validity), because the former specifies a minimum diagnostic criterion of performance, whereas the latter only entails a comparative ordering of performance without a criterion being necessarily meet. In that sense, accuracy is a necessary but insufficient condition for robustness.
A breakdown of MSE values by estimator and sample size is provided in Figure S6 of the supplementary documentation.
Serlin (2000) proposed a more demanding robustness level of \(\Delta =0.0125\) as a compromise between Cochran’s (1952) \(\Delta =0.02\) criterion and Bradley’s (1978) stringent \(\Delta =0.005\) criterion. If Serlin’s more rigorous criterion had been used in the current investigation, then 59% of HC3 intervals would still be inferred as being robust (which remains highest among all HC estimators). In contrast, normal theory intervals would drop to 29% and ADF intervals would be robust in 20% of instances.
The supplementary documentation (Section S7) contains a trellis plot for the disaggregation of robustness by the orientation angle of regression coefficients. Although the degrading effect of \(\angle {\varvec{\upbeta } }_{90^{\circ }} \) on \({ CI}_{.95}\)(N) is obvious, and to a lesser extent on the ADF estimator, this factor had no consistent effect on the HC estimators.
References
Arminger, G., & Schoenberg, R. J. (1989). Pseudo-maximum likelihood estimation and a test for misspecification in mean and covariance structure models. Psychometrika, 54, 409–425. doi:10.1007/BF02294626.
Bentler, P. M., & Lee, S. Y. (1983). Covariance structures under polynomial constraints: Applications to correlation and alpha-type structure models. Journal of Educational Statistics, 8, 207–222. doi:10.3102/10769986008003207.
Bentler, P. M., & Wu, E. J. C. (2000–2008). EQS version 6.2 [Computer software]. Encino, CA.: Multivariate Software, Inc.
Bradley, J. V. (1978). Robustness? British Journal of Mathematical and Statistical Psychology, 31, 144–152. doi:10.1111/j.2044-8317.1978.tb00581.x.
Browne, M. W. (1984). Asymptotically distribution-free methods for the analysis of covariance structures. British Journal of Mathematical and Statistical Psychology, 37, 62–83. doi:10.1111/j.2044-8317.1984.tb00789.x.
Browne, M. W., Mels, G., & Cowan, M. (2010). Path analysis: RAMONA in SYSTAT version 13 [software]. Chicago, IL: Systat Software Inc.
Chan, W., Yung, Y.-F., & Bentler, P. M. (1995). A note on using an unbiased weight matrix in the ADF test statistic. Multivariate Behavioral Research, 30, 453–459. doi:10.1207/s15327906mbr3004_1.
Cochran, W. G. (1952). The \(\chi ^{2}\) test of goodness of fit. Annals of Mathematical Statistics, 23, 315–345. doi:10.1214/aoms/1177729380.
Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/ correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.
Cribari-Neto, F. (2004). Asymptotic inference under heteroskedasticity of unknown form. Computational Statistics and Data Analysis, 45, 215–233. doi:10.1016/S0167-9473(02)00366-3.
Cribari-Neto, F., & Da Silva, W. B. (2011). A new heteroskedasticity-consistent covariance matrix estimator for the linear regression model. Advances in Statistical Analysis, 95, 129–146. doi:10.1007/s10182-010-0141-2.
Cribari-Neto, F., Souza, T. C., & Vasconcellos, K. L. P. (2007). Inference under heteroskedasticity and leveraged data. Communications in Statistics: Theory and Methods, 36, 1877–1888. doi:10.1080/03610920601126589.
Cudeck, R. (1989). Analysis of correlation matrices using covariance structure models. Psychological Bulletin, 105, 317–327. doi:10.1037/0033-2909.105.2.317.
Efron, B., & Tibshirani, R. (1998). An introduction to the bootstrap. Boca Raton, FL: Chapman and Hall.
Finney, S. J., & DiStefano, C. (2013). Nonnormal and categorical data in structural equation modeling. In G. R. Hancock & R. O. Muller (Eds.), Structural equation modeling: A second course (2nd ed., pp. 439–492). Charlotte, NC: Information Age Publishing Inc.
Hayes, A. F., & Cai, L. (2007). Using heteroskedasticity-consistent standard error estimators in OLS regression: An introduction and software implementation. Behavior Research Methods, 39, 709–722. doi:10.3758/BF03192961.
Headrick, T. C. (2002). Fast fifth-order polynomial transforms for generating univariate and multivariate nonnormal distributions. Computational Statistics and Data Analysis, 40, 865–711. doi:10.1016/S0167-9473(02)00072-5.
Headrick, T. C. (2012). Statistical simulation: Power method polynomials and other transformations. Boca Raton, FL: CRC Press.
Headrick, T. C., Sheng, Y., & Hodis, F. (2007). Numerical computing and graphics for the power method transformation using mathematica. Journal of Statistical Software, 19(2), 1–17. doi:10.18637/jss.v019.i03.
Hoogland, J. J., & Boomsma, A. (1998). Robustness studies in covariance structure modeling: An overview and meta-analysis. Sociological Methods and Research, 26, 329–367. doi:10.1177/0049124198026003003.
Hu, L.-T., Bentler, P. M., & Kano, Y. (1992). Can test statistics in covariance structure analysis be trusted? Psychological Bulletin, 112, 351–362. doi:10.1037/0033-2909.112.2.351.
Jones, J. A., & Waller, N. G. (2013). Computing confidence intervals for standardized regression coefficients. Psychological Methods, 18, 435–453. doi:10.1037/a0033269.
Jones, J. A., & Waller, N. G. (2015). The normal-theory and asymptotic distribution-free (ADF) covariance matrix of standardized regression coefficients: Theoretical extensions and finite sample behavior. Psychometrika, 80, 365–378. doi:10.1007/S11336-013-9380-Y.
Kelley, K. (2007). Confidence intervals for standardized effect sizes: Theory, application, and implementation. Journal of Statistical Software, 20, 1–24.
Kowalchuk, R., & Headrick, T. C. (2010). Simulating multivariate g-and-h distributions. British Journal of Mathematical and Statistical Psychology, 63, 63–74. doi:10.1348/000711009X423067.
Long, J. S., & Ervin, L. H. (2000). Using heteroskedasticity consistent standard errors in the linear regression model. The American Statistician, 54, 217–224. doi:10.1080/00031305.2000.10474549.
MacKinnon, J. G., & White, H. (1985). Some heteroskedasticity consistent covariance matrix estimators with improved finite sample properties. Journal of Econometrics, 35, 305–325. doi:10.1016/0304-4076(85)90158-7.
Magnus, J. R., & Neudecker, H. (2007). Matrix differential calculus with applications in statistics and economics (3rd ed.). Chichester: Wiley.
Mathworks, Inc. (2015). MATLAB, Version 8.5.0.197613 (R2015a) [Computer software]. Natick, MA: Mathworks, Inc.
Muthén, B. O., & Asparouhov, T. (2012). Bayesian structural equation modeling: A more flexible representation of substantive theory. Psychological Methods, 17, 313–335. doi:10.1037/a0026802.
Muthén, L. K., & Muthén, B. O. (1998–2012). Mplus user’s guide (7th ed.) [Computer software]. Los Angeles, CA: Muthén & Muthén.
Ng, M., & Wilcox, R. R. (2009). Level robust methods based on the least squares regression estimator. Journal of Modern Applied Statistical Methods, 8(Issue 2, Article 5), 384–395. From http://digitalcommons.wayne.edu/jmasm/vol8/iss2/5
Nel, D. G. (1980). On matrix differentiation in statistics. South African Statistics Journal, 14, 137–193.
Olvera Astivia, O. L., & Zumbo, B. (2015). A cautionary note on the use of the Vale and Maurelli method to generate multivariate, nonnormal data for simulation purposes. Educational and Psychological Measurement, 75, 541–567. doi:10.1177/0013164414548894.
R Development Core Team. (2015). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org
Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48, 1–36.
Rozeboom, W. W. (1966). Foundations of the theory of prediction. Homewood, Illinois: Dorsey Press.
Sampson, A. R. (1974). A tale of two regressions. Journal of the American Statistical Association, 69, 682–689. doi:10.1080/01621459.1974.10480189.
Savalei, V. (2014). Understanding robust corrections in structural equation modeling. Structural Equation Modeling, 21, 149–160. doi:10.1080/10705511.2013.824793.
Serlin, R. C. (2000). Testing for robustness in Monte Carlo studies. Psychological Methods, 5, 230–240. doi:10.1037/1082-989X.5.2.230.
StataCorp., (2015). Stata programming reference manual: Release 14. College Station, TX: StataCorp LP.
Steiger, J. H. (2015). SEPath: Structural equation modelling program (version 12.7) [Computer software]. Tulsa, OK: StatSoft, Inc.
Stevens, J. P. (2009). Applied multivariate statistics for the social sciences (5th ed.). New York: Psychology Press Taylor & Francis.
Vale, C. D., & Maurelli, V. A. (1983). Simulating multivariate nonnormal distributions. Psychometrika, 48, 465–471. doi:10.1007/BF02293687.
Waller, N. G., & Jones, J. A. (2010). Correlation weights in multiple regression. Psychometrika, 75, 58–69. doi:10.1007/s11336-009-9127-y.
Waller, N. G., & Jones, J. A. (2011). Investigating the performance of alternate regression weights by studying all possible criteria in regression models with a fixed set of predictors. Psychometrika, 76, 410–439. doi:10.1007/s11336-011-9209-5.
Waller, N. G., & Jones, J. A. (2015). Fungible: Fungible coefficients and Monte Carlo functions (R package version 1.2). Retrieved from https://cran.r-project.org/
Wasserman, L. (2003). All of statistics: A concise course in statistical inference. New York: Springer.
White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48, 817–838. doi:10.2307/1912934.
White, H. (1982). Maximum likelihood estimation of misspecified models. Econometrica, 50, 1–25. doi:10.2307/1912526.
Yuan, K.-H., & Bentler, P. M. (1998a). Robust mean and covariance structure analysis. British Journal of Mathematical and Statistical Psychology, 51, 63–88. doi:10.1111/j.2044-8317.1998.tb00667.x.
Yuan, K.-H., & Bentler, P. M. (1998b). Structural equation modeling with robust covariances. In A. E. Raftery (Ed.), Sociological Methodology 1998 (pp. 363–396). Boston, MA: Blackwell. doi:10.1111/0081-1750.00052
Yuan, K.-H., & Chan, W. (2011). Biases and standard errors of standardized regression coefficients. Psychometrika, 76, 670–690. doi:10.1007/S11336-011-9224-6.
Yuan, K.-H., & Hayashi, K. (2006). Standard errors in covariance structure models: Asymptotics versus bootstrap. British Journal of Mathematical and Statistical Psychology, 59, 397–417. doi:10.1348/000711005X85896.
Yung, Y.-F., & Bentler, P. M. (1994). Bootstrap-corrected ADF test statistics in covariance structure analysis. British Journal of Mathematical and Statistical Psychology, 47, 63–84. doi:10.1111/j.2044-8317.1994.tb01025.x.
Zeileis, A. (2004). Econometric computing with HC and HAC covariance matrix estimators. Journal of Statistical Software, 16, 1–17. Retrieved from http://www.jstatsoft.org/v11/i10
Acknowledgements
All MATLAB functions used in this research are available from the author by email request. This research was supported by funding from the Australian Research Council (Project Grants DP120101402 and LP130100314) and from the National Health and Medical Research Council, Australia (Project Grants 1027076 and APP1082668).
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Appendix
Appendix
Let the parameter vector for the specification in Equation 24 be ordered as
which may be presented in augmented form as
where \(\hbox {vecp}(\mathbf{P}_{\varvec{x}})\) is an \(s\times 1\) vector of correlations below the diagonal among independent variables in \(\mathbf{P}_{\varvec{x}} \), with \(s=p\times (p-1)/2\). Let \(\mathbf{D}_\sigma =\hbox {Diag}\left[ {\sigma _y ,\sigma (x_1 ),\ldots ,\sigma (x_p )} \right] \) be a \(q\times q\) diagonal matrix of standard deviations. Let \(\mathbf{J}_k^{(i,j)} \) in general denote a single-entry \(k\times k\) matrix in which all elements are 0 except the (i, j) element that equals 1. Likewise, let in general be a \(k\times k\) matrix in which all elements are 0 except the (i, j) and (j, i) element that each equal 1.
Given the above definitions, and others in the main body of the paper, the \(q^{*}\times \;q^{*}\) Jacobian matrix of partial derivatives of \(\hbox {vech}\left[ {{\varvec{\Sigma }}({\varvec{\uptheta }}_\beta )} \right] \) with respect to \({\varvec{\uptheta }}_\beta \) can be given in terms of augmented parts as follows:
-
1.
The partial derivative of the jth standardized regression coefficient in \({\varvec{\upbeta } }\) is given by
$$\begin{aligned} \frac{\partial \hbox {vech}\left[ {{\varvec{\Sigma }}({\varvec{\uptheta }}_\beta )} \right] }{\partial \beta _j }=\hbox {vech}\left[ {\mathbf{D}_\sigma \mathbf{Z}_c \mathbf{D}_\sigma } \right] , \end{aligned}$$where \(\mathbf{Z}_c \) is a \(q\times q\) null matrix except where the initial row elements \(\mathbf{Z}_C (1,2:q)\) is set equal to the jth row of \(\mathbf{P}_{\varvec{x}}\) and the column elements \(\mathbf{Z}_C (2:q,1)\) are equal to the jth column of \(\mathbf{P}_{\varvec{x}} \).
-
2.
The partial derivative of \(\sigma _y \) is given by
$$\begin{aligned} \frac{\partial \hbox {vech}\left[ {{\varvec{\Sigma }}({\varvec{\uptheta }}_\beta )} \right] }{\partial \sigma _y }=\hbox {vech}\left[ {\mathbf{J}_q^{(1,1)} {\mathbf{P}}_{y{\varvec{x}}} \mathbf{D}_\sigma +\mathbf{D}_\sigma {\mathbf{P}}_{y{\varvec{x}}} \mathbf{J}_q^{(1,1)} } \right] . \end{aligned}$$ -
3.
The partial derivative of the standard deviation for the jth independent variable in \({\varvec{\upsigma }}_{\varvec{x}}\) is given by
$$\begin{aligned} \frac{\partial \hbox {vech}\left[ {{\varvec{\Sigma }}({\varvec{\uptheta }}_\beta )} \right] }{\partial \sigma (x_j )}=\hbox {vech}\left[ {\mathbf{J}_q^{(u,u)} {\mathbf{P}}_{y{\varvec{x}}} \mathbf{D}_\sigma +\mathbf{D}_\sigma {\mathbf{P}}_{y{\varvec{x}}} \mathbf{J}_q^{(u,u)} } \right] , \end{aligned}$$where \(u=j+1\).
-
4.
Finally, the partial derivative of the correlation \(r(x_j ,x_i )\), for \(j=2,\ldots ,p\) and \(i<j\), among the set of independent variables is given by
$$\begin{aligned} \frac{\partial \hbox {vech}\left[ {{\varvec{\Sigma }}({\varvec{\uptheta }}_\beta )} \right] }{\partial \hbox {r}(x_i ,x_j )}=\hbox {vech}\left[ {\mathbf{D}_\sigma \mathbf{C}_q \mathbf{D}_\sigma } \right] , \end{aligned}$$where the \(q\times q\) matrix \(\mathbf{C}_q \) is partitioned as
with \(u=j+1\) and \(v=i+1\).
Rights and permissions
About this article
Cite this article
Dudgeon, P. Some Improvements in Confidence Intervals for Standardized Regression Coefficients. Psychometrika 82, 928–951 (2017). https://doi.org/10.1007/s11336-017-9563-z
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11336-017-9563-z