^{1}

^{2}

^{*}

^{3}

^{1}

^{2}

^{3}

Edited by: Holmes Finch, Ball State University, USA

Reviewed by: Yanyan Sheng, Southern Illinois University, USA; José Manuel Reales, Universidad Nacional de Educación a Distancia, Spain

*Correspondence: Italo Trizano-Hermosilla

This article was submitted to Quantitative Psychology and Measurement, a section of the journal Frontiers in Psychology

This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

The Cronbach's alpha is the most widely used method for estimating internal consistency reliability. This procedure has proved very resistant to the passage of time, even if its limitations are well documented and although there are better options as omega coefficient or the different versions of glb, with obvious advantages especially for applied research in which the ítems differ in quality or have skewed distributions. In this paper, using Monte Carlo simulation, the performance of these reliability coefficients under a one-dimensional model is evaluated in terms of skewness and no tau-equivalence. The results show that omega coefficient is always better choice than alpha and in the presence of skew items is preferable to use omega and glb coefficients even in small samples.

The α coefficient is the most widely used procedure for estimating reliability in applied research. As stated by Sijtsma (

The assumption of uncorrelated errors (the error score of any pair of items is uncorrelated) is a hypothesis of Classical Test Theory (Lord and Novick,

The assumption of tau-equivalence (i.e., the same true score for all test items, or equal factor loadings of all items in a factorial model) is a requirement for α to be equivalent to the reliability coefficient (Cronbach,

The requirement for multivariant normality is less known and affects both the puntual reliability estimation and the possibility of establishing confidence intervals (Dunn et al.,

Considering the abundant literature on the limitations and biases of the α coefficient (Revelle and Zinbarg,

The difficulty of estimating the

McDonald (_{t} coefficient for estimating reliability from a factorial analysis framework, which can be expressed formally as:

Where λ_{j} is the loading of item _{t} coefficient, by including the lambdas in its formulas, is suitable both when tau-equivalence (i.e., equal factor loadings of all test items) exists (ω_{t} coincides mathematically with α), and when items with different discriminations are present in the representation of the construct (i.e., different factor loadings of the items: congeneric measurements). Consequently ω_{t} corrects the underestimation bias of α when the assumption of tau-equivalence is violated (Dunn et al.,

When correlation exists between errors, or there is more than one latent dimension in the data, the contribution of each dimension to the total variance explained is estimated, obtaining the so-called hierarchical ω (ω_{h}) which enables us to correct the worst overestimation bias of α with multidimensional data (see Tarkkonen and Vehkalahti, _{h} and ω_{t} are equivalent in unidimensional data, so we will refer to this coefficient simply as ω.

Sijtsma (_{x} = _{t} + _{e})—an inter-item covariance matrix for observed item scores _{x}. It breaks down into two parts: the sum of the inter-item covariance matrix for item true scores _{t}; and the inter-item error covariance matrix _{e} (ten Berge and Sočan,

where _{e}) refers to the trace of the inter-item error covariance matrix which it has proved so difficult to estimate. One solution has been to use factorial procedures such as Minimum Rank Factor Analysis (a procedure known as glb.fa). More recently the GLB algebraic (GLBa) procedure has been developed from an algorithm devised by Andreas Moltner (Moltner and Revelle,

Despite its theoretical strengths, GLB has been very little used, although some recent empirical studies have shown that this coefficient produces better results than α (Lila et al.,

Considering the coefficients defined above, and the biases and limitations of each, the object of this work is to evaluate the robustness of these coefficients in the presence of asymmetrical items, considering also the assumption of tau-equivalence and the sample size.

The data were generated using R (R Development Core Team,

where _{ij} is the simulated response of subject _{jk} is the loading of item _{k} is the latent factor generated by a standardized normal distribution (mean 0 and variance 1), and _{j} is the random measurement error of each item also following a standardized normal distribution.

Skewed items: Standard normal _{ij} were transformed to generate non-normal distributions using the procedure proposed by Headrick (

The coefficients implemented by Sheng and Sheng (_{0} = −0.446924, _{1} = 1.242521, _{2} = 0.500764, _{3} = −0.184710, _{4} = −0.017947, _{5} = 0.003159.

To assess the performance of the reliability coefficients (α, ω, GLB and GLBa) we worked with three sample sizes (250, 500, 1000), two test sizes: short (6 items) and long (12 items), two conditions of tau-equivalence (one with tau-equivalence and one without, i.e., congeneric) and the progressive incorporation of asymmetrical items (from all the items being normal to all the items being asymmetrical). In the short test the reliability was set at 0.731, which in the presence of tau-equivalence is achieved with six items with factor loadings = 0.558; while the congeneric model is obtained by setting factor loadings at values of 0.3, 0.4, 0.5, 0.6, 0.7, and 0.8 (see Appendix I). In the long test of 12 items the reliability was set at 0.845 taking the same values as in the short test for both tau-equivalence and the congeneric model (in this case there were two items for each value of lambda). In this way 120 conditions were simulated with 1000 replicas in each case.

The main analyses were carried out using the

In order to evaluate the accuracy of the various estimators in recovering reliability, we calculated the Root Mean Square of Error (

where

In both indices, the greater the value, the greater the inaccuracy of the estimator, but unlike RMSE, the bias may be positive or negative; in this case additional information would be obtained as to whether the coefficient is underestimating or overestimating the simulated reliability parameter. Following the recommendation of Hoogland and Boomsma (

The principal results can be seen in Table

NORMALITY | |||||||||||

0 items | 0.0 | 250 | tau | 0.03 | 0.03 | 0.04 | −0.20 | 0.00 | 3.50 | ||

cong | 0.03 | 0.02 | 0.04 | −1.60 | −0.10 | 3.40 | |||||

500 | tau | 0.02 | 0.02 | 0.03 | 0.00 | 0.00 | 2.50 | ||||

cong | 0.02 | 0.02 | 0.03 | −1.50 | 0.00 | 2.50 | |||||

1000 | tau | 0.01 | 0.01 | 0.02 | −0.10 | −0.10 | 4.90 | 1.70 | |||

cong | 0.02 | 0.01 | 0.02 | −1.50 | −0.10 | 1.70 | |||||

2 items | 0.3 | 250 | tau | 0.04 | 0.03 | − |
− |
2.70 | −0.80 | ||

cong | 0.04 | 0.03 | − |
−3.00 | 3.90 | 0.90 | |||||

500 | tau | 0.04 | 0.03 | − |
−4.90 | 1.80 | −1.90 | ||||

cong | 0.03 | 0.04 | 0.02 | − |
−2.80 | 3.20 | −0.10 | ||||

1000 | tau | 0.03 | 0.03 | − |
− |
1.00 | −2.70 | ||||

cong | 0.03 | 0.04 | 0.02 | − |
−2.90 | 2.40 | −1.00 | ||||

4 items | 0.6 | 250 | tau | 0.04 | − |
− |
−0.80 | −4.70 | |||

cong | 0.04 | − |
− |
0.20 | −3.50 | ||||||

500 | tau | 0.04 | − |
− |
−1.60 | − |
|||||

cong | 0.03 | − |
− |
−0.30 | −4.60 | ||||||

1000 | tau | 0.04 | − |
− |
−2.50 | − |
|||||

cong | 0.03 | 0.04 | − |
− |
0.80 | −3.10 | |||||

All items | 0.9 | 250 | tau | − |
- |
−3.00 | − |
||||

cong | − |
− |
−2.50 | − |
|||||||

500 | tau | − |
− |
−4.10 | − |
||||||

cong | − |
− |
−3.40 | − |
|||||||

1000 | tau | − |
- |
− |
− |
||||||

cong | − |
− |
−4.10 | − |

NORMALITY | |||||||||||

0 items | 0.0 | 250 | tau | 0.02 | 0.02 | 0.04 | −0.10 | −0.10 | 4.80 | 3.90 | |

cong | 0.02 | 0.01 | 0.04 | −0.90 | −0.10 | 4.40 | 3.80 | ||||

500 | tau | 0.01 | 0.01 | 0.04 | 0.03 | −0.10 | −0.10 | 4.20 | 2.80 | ||

cong | 0.01 | 0.01 | 0.04 | 0.03 | −0.80 | −0.10 | 3.70 | 2.70 | |||

1000 | tau | 0.01 | 0.01 | 0.04 | 0.02 | 0.00 | 0.00 | 3.70 | 2.00 | ||

cong | 0.01 | 0.01 | 0.03 | 0.02 | −0.80 | 0.00 | 3.10 | 2.00 | |||

2 items | 0.2 | 250 | tau | 0.03 | 0.02 | 0.04 | 0.03 | −1.90 | −1.70 | 3.60 | 2.60 |

cong | 0.03 | 0.02 | 0.04 | 0.03 | −2.00 | −1.00 | 3.70 | 3.10 | |||

500 | tau | 0.02 | 0.02 | 0.03 | 0.02 | −1.80 | −1.70 | 2.90 | 1.40 | ||

cong | 0.02 | 0.01 | 0.03 | 0.02 | −2.00 | −1.00 | 3.00 | 1.90 | |||

1000 | tau | 0.02 | 0.02 | 0.03 | 0.01 | −1.80 | −1.70 | 2.40 | 0.60 | ||

cong | 0.02 | 0.01 | 0.03 | 0.01 | −1.90 | −1.00 | 2.30 | 1.10 | |||

4 items | 0.3 | 250 | tau | 0.04 | 0.04 | 0.03 | 0.02 | −3.60 | −3.50 | 2.40 | 1.40 |

cong | 0.04 | 0.03 | 0.03 | 0.02 | −3.90 | −2.80 | 2.50 | 1.80 | |||

500 | tau | 0.04 | 0.04 | 0.02 | 0.01 | −3.50 | −3.40 | 1.70 | 0.10 | ||

cong | 0.04 | 0.03 | 0.02 | 0.01 | −3.80 | −2.70 | 1.70 | 0.60 | |||

1000 | tau | 0.04 | 0.04 | 0.02 | 0.01 | −3.50 | −3.40 | 1.00 | −0.80 | ||

cong | 0.04 | 0.03 | 0.02 | 0.01 | −3.80 | −2.70 | 1.00 | −0.30 | |||

6 items | 0.5 | 250 | tau | 0.02 | 0.02 | − |
− |
1.30 | 0.20 | ||

cong | 0.02 | 0.02 | − |
− |
1.00 | 0.30 | |||||

500 | tau | 0.02 | 0.02 | − |
− |
0.40 | −1.20 | ||||

cong | 0.02 | 0.02 | − |
− |
−0.10 | −1.10 | |||||

1000 | tau | 0.02 | 0.02 | − |
− |
−0.40 | −2.20 | ||||

cong | 0.02 | 0.02 | − |
− |
−0.80 | −2.10 | |||||

8 items | 0.6 | 250 | tau | 0.02 | 0.02 | − |
− |
0.20 | −1.00 | ||

cong | 0.02 | 0.02 | − |
− |
0.20 | −0.50 | |||||

500 | tau | 0.02 | 0.03 | − |
− |
−0.70 | −2.50 | ||||

cong | 0.02 | 0.02 | − |
− |
−0.80 | −1.90 | |||||

1000 | tau | 0.02 | 0.04 | − |
− |
−1.50 | −3.60 | ||||

cong | 0.02 | 0.03 | − |
− |
−1.70 | −3.00 | |||||

10 items | 0.8 | 250 | tau | 0.03 | 0.03 | − |
− |
−0.70 | −2.00 | ||

cong | 0.02 | 0.03 | − |
− |
−0.90 | −1.60 | |||||

500 | tau | 0.03 | 0.04 | − |
− |
−1.70 | −3.60 | ||||

cong | 0.03 | 0.04 | − |
− |
−1.60 | −3.10 | |||||

1000 | tau | 0.03 | − |
− |
−2.40 | −4.80 | |||||

cong | 0.03 | 0.04 | − |
− |
−2.40 | −4.30 | |||||

All items | 1.0 | 250 | tau | 0.03 | 0.04 | − |
− |
−1.40 | −2.80 | ||

cong | 0.03 | 0.03 | − |
− |
−1.30 | −2.30 | |||||

500 | tau | 0.03 | − |
− |
−2.30 | −4.60 | |||||

cong | 0.03 | 0.04 | − |
− |
−2.30 | −3.90 | |||||

1000 | tau | 0.04 | − |
− |
−3.20 | − |
|||||

cong | 0.04 | − |
− |
−3.20 | − |

Only under conditions of tau-equivalence and normality (skewness < 0.2) is it observed that the α coefficient estimates the simulated reliability correctly, like ω. In the congeneric condition ω corrects the underestimation of α. Both GLB and GLBa present a positive bias under normality, however GLBa shows approximatively ½ less % bias than GLB (see Table

In asymmetrical conditions, we see in Table

In this study four factors were manipulated: tau-equivalence or congeneric model, sample size (250, 500, and 1000), the number of test items (6 and 12) and the number of asymmetrical items (from 0 asymmetrical items to all the items being asymmetrical) in order to evaluate robustness to the presence of asymmetrical data in the four reliability coefficients analyzed. These results are discussed below.

In conditions of tau-equivalence, the α and ω coefficients converge, however in the absence of tau-equivalence (congeneric), ω always presents better estimates and smaller RMSE and % bias than α. In this more realistic condition therefore (Green and Yang,

Turning to sample size, we observe that this factor has a small effect under normality or a slight departure from normality: the RMSE and the bias diminish as the sample size increases. Nevertheless, it may be said that for these two coefficients, with sample size of 250 and normality we obtain relatively accurate estimates (Tang and Cui,

For the test size we generally observe a higher RMSE and bias with 6 items than with 12, suggesting that the higher the number of items, the lower the RMSE and the bias of the estimators (Cortina,

When we look at the effect of progressively incorporating asymmetrical items into the data set, we observe that the α coefficient is highly sensitive to asymmetrical items; these results are similar to those found by Sheng and Sheng (

Considering that in practice it is common to find asymmetrical data (Micceri,

These results are limited to the simulated conditions and it is assumed that there is no correlation between errors. This would make it necessary to carry out further research to evaluate the functioning of the various reliability coefficients with more complex multidimensional structures (Reise,

When the total test scores are normally distributed (i.e., all items are normally distributed) ω should be the first choice, followed by α, since they avoid the overestimation problems presented by GLB. However, when there is a low or moderate test skewness GLBa should be used. GLB is recommended when the proportion of asymmetrical items is high, since under these conditions the use of both α and ω as reliability estimators is not advisable, whatever the sample size.

Development of the idea of research and theoretical framework (IT, JA). Construction of the methodological framework (IT, JA). Development of the R language syntax (IT, JA). Data analysis and interpretation of data (IT, JA). Discussion of the results in light of current theoretical background (JA, IT). Preparation and writing of the article (JA, IT). In general, both authors have contributed equally to the development of this work.

The first author disclosed receipt of the following financial support for the research, authorship,

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

R syntax to estimate reliability coefficients from Pearson's correlation matrices. The correlation values outside the diagonal are calculated by multiplying the factor loading of the items: (1) tau-equivalent model they are all equal to 0.3114 (λ_{i}λ_{j} = 0.558 × 0.558 = 0.3114) and (2) congeneric model they vary as a function of the different factor loading (e.g., the matrix element a_{1, 2} = λ_{1}λ_{2} = 0.3 × 0.4 = 0.12). In both examples the true reliability is 0.731.

> library(psych)

> library(Rcsdp)

> Cr <-matrix(c(1.00, 0.3114, 0.3114, 0.3114, 0.3114, 0.3114,

0.3114, 1.00, 0.3114, 0.3114, 0.3114, 0.3114,

0.3114, 0.3114, 1.00, 0.3114, 0.3114, 0.3114,

0.3114, 0.3114, 0.3114, 1.00, 0.3114, 0.3114,

0.3114, 0.3114, 0.3114, 0.3114, 1.00, 0.3114,

0.3114, 0.3114, 0.3114, 0.3114, 0.3114, 1.00),

ncol = 6)

> omega(Cr,1)$alpha # standardized Cronbach's α

[1] 0.731

> omega(Cr,1)$omega.tot # coefficient ω total

[1] 0.731

> glb.fa(Cr)$glb # GLB factorial procedure

[1] 0.731

> glb.algebraic(Cr)$glb # GLB algebraic procedure

[1] 0.731

_{1} _{2} _{3} _{4} _{5} _{6}

> Cr <-matrix(c(1.00, 0.12, 0.15, 0.18, 0.21, 0.24,

0.12, 1.00, 0.20, 0.24, 0.28, 0.32,

0.15, 0.20, 1.00, 0.30, 0.35, 0.40,

0.18, 0.24, 0.30, 1.00, 0.42, 0.48,

0.21, 0.28, 0.35, 0.42, 1.00, 0.56,

0.24, 0.32, 0.40, 0.48, 0.56, 1.00),

ncol = 6)

> omega(Cr,1)$alpha # standardized Cronbach's α

[1] 0.717

> omega(Cr,1)$omega.tot # coefficient ω total

[1] 0.731

> glb.fa(Cr)$glb # GLB factorial procedure

[1] 0.754

> glb.algebraic(Cr)$glb # GLB algebraic procedure

[1] 0.731