﻿ Reliability and separation of measures

# Reliability and separation of measures

Winsteps report reliability and separation statistics treating the sample of measures as the population. If it is not the entire population, then the reliability and separation are slightly higher than the reported values.

Usually person and item reliability and separation have different applications and implications.

Person separation is used to classify people. Low person separation (< 2, person reliability < 0.8 )with a relevant person sample implies that the instrument may not be not sensitive enough to distinguish between high and low performers. More items may be needed.

Item separation is used to verify the item hierarchy. Low item separation (< 3 = high, medium, low item difficulties, item reliability < 0.9) implies that the person sample is not large enough to confirm the item difficulty hierarchy (= construct validity) of the instrument.

Reliability (separation index) means "reproducibility of relative measure location". It does not report on the quality of the data. So "high reliability" (of persons or items) means that there is a high probability that persons (or items) estimated with high measures actually do have higher measures than persons (or items) estimated with low measures. If you want high reliability, you need a wide sample and/or low measurement error. So, if you want high person (test) reliability, you need a person sample with a large ability (or whatever) range and/or an instrument with many items (or long rating scales). If you want high item reliability, you need a test with a large item difficulty range and/or a large sample of persons. Usually low item reliability is because the person sample size is too small to establish a reproducible item difficulty hierarchy.

Missing data: if some persons have missing observations, these can considerably reduce precision, and so lower reliability estimates.  Suggestion: omit person-records with missing data when estimating reliabilities.

Person (sample, test) reliability depends chiefly on

1) Sample ability variance. Wider ability range = higher person reliability.

2) Length of test (and rating scale length). Longer test = higher person reliability

3) Number of categories per item. More categories = higher person reliability

4) Sample-item targeting. Better targeting = higher person reliability

It is independent of sample size. It is largely uninfluenced by model fit.

In general, Test Reliability reported by Classical Test Theory (Cronbach Alpha, KR-20) is higher than Rasch Reliability. Rasch Reliability is higher than 3-PL IRT Reliability.

Item reliability depends chiefly on

1) Item difficulty variance. Wide difficulty range = high item reliability

2) Person sample size. Large sample = high item reliability

It is independent of test length. It is largely uninfluenced by model fit.

Tentative guidelines:

Person reliability: Does your test discriminate the sample into enough levels for your purpose? 0.9 = 3 or 4 levels. 0.8 = 2 or 3 levels. 0.5 = 1 or 2 levels.

Item reliability: Low reliability means that your sample is not big enough to precisely locate the items on the latent variable.

Rater reliability: Low "separation" reliability is better, because we want raters to be reliably the same, not reliably different.

The Winsteps "person reliability" is equivalent to the traditional "test" reliability. Low values indicate a narrow range of person measures, or a small number of items. To increase person reliability, test persons with more extreme abilities (high and low), lengthen the test. Improving the test targeting may help slightly.

The Winsteps "item reliability" has no traditional equivalent. Low values indicate a narrow range of item measures, or a small sample. To increase "item reliability", test more people. In general, low item reliability means that your sample size is too small for stable item estimates based on the current data. If you have anchored values, then it is the item reliability of the source from which the anchor values emanate which is crucial, not the current sample.

The "model" person reliability (including measures for extreme scores) is an upper bound to this value, when persons are ordered by measures.

The "real" person reliability (including measures for extreme scores) is a lower bound to this value, when persons are ordered by measures

The traditional "test reliability", as defined by Charles Spearman in 1904, etc., is the "true person variance / observed person variance" for this sample on these test items. So it is really a "person sample reliability" rather than a "test reliability", where reliability = reproducibility of person ordering. The "true person variance" cannot be known, but it can be approximated. KR-20 approximates it by summarizing item point-biserials. Cronbach Alpha approximates it with an analysis of variance. Winsteps approximates it using the measure standard errors.

The separation coefficient and reliability computations are computed with and without any elements with extreme measures. Since the measures for extreme scores are imprecise, reliability statistics which include extreme scores are often lower than their non-extreme equivalents. Conventional computation of a reliability coefficient (KR-20, Cronbach Alpha) includes persons with extreme scores. The classical reliability computation includes extreme scores (if any) is the conventional reliability, and usually produces an estimate between the MODEL and REAL values, closer to the MODEL or even above it.

KR-20 value is an estimate of the value when persons are ordered by raw scores. CRONBACH ALPHA (KR-20) KID RAW SCORE RELIABILITY is the conventional "test" reliability index. It reports an approximate test reliability based on the raw scores of this sample. It is only reported for complete data. An apparent paradox is that extreme scores have perfect precision, but extreme measures have perfect imprecision.

Winsteps computes upper and lower boundary values for the True Reliability. The lower boundary is the Real Reliability. The upper boundary is the Model Reliability. The unknowable True Reliability lies somewhere between these two. As contradictory sources of noise are remove from the data, the True Reliability approaches the Model Reliability

Cronbach Alpha and KR-20 Reliability

Here is a check on the computations. Guilford reports 0.81. Winsteps reports 0.82. The difference is probably computational precision and rounding error.

Title = "Guilford Table 17.2. His Cronbach Alpha = KR-20 = 0.81"

ni=8

item1=1

name1=1

&END

END LABELS

00000000

10000000

10100000

11001000

01010010

11101010

11111100

11111100

11110101

11111111

Conventionally, only a Person ("Test") Reliability is reported. The relationship between raw-score-based reliability (i.e., KR-20, Cronbach Alpha) and measure-based reliability is complex, see www.rasch.org/rmt/rmt113l.htm - in general, Cronbach Alpha overestimates reliability, Rasch underestimates it. So, when it is likely that the Rasch reliability will be compared with conventional KR-20 or Cronbach Alpha reliabilities (which are always computed assuming the data match their assumptions), then include extreme persons and report the higher Rasch reliability, the "Model" reliability, computed on the assumption that all unexpectedness in the data is in accord with Rasch model predictions. The big differences between Score and Measure reliabilities occur when

(a) there are extreme scores. These increase score reliability, but decrease measure reliability.

(b) missing data. Missing data always decreases measure reliability. If the missing data are imputed at their expected values (in order to make conventional reliability formulas computable), they increase score reliability. Winsteps attempts to adjust the raw-score reliability for this inflation in the raw-score reliability, but can only do the adjustment in an approximate way.

Winsteps also reports an item reliability, "true item variance / observed item variance". When this value is low, it indicates that the sample size may be too small for stable comparisons between items.

Anchored values are treated as though they are the "true values" of the MLE estimates. Their local standard errors are estimated using the current data in the same way as unanchored MLE standard error estimates. It is the measures (anchored or unanchored) and local standard errors that are used in the reliability computations. If you wish to compute reliabilities using different standard error estimates (e.g., the ones when the anchor values were generated), then please perform a separate reliability computation (using Excel).

You can easily check the Winsteps reliability estimate computation yourself.

Compute the STDEVP standard deviation of the person measures. Square it. This is the "Observed variance".

"Model" Reliability: Take the standard ERROR column. Square each entry. Sum the squared entries. Divide that sum by the count of entries. This is the "Model Error variance" estimate. Then,

Model Reliability = True Variance / Observed Variance = (Observed Variance - Model Error Variance) / Observed Variance.

"Real" Reliability: Take the standard ERROR column. Square each entry, SE². In another column, put SE²*Maximum [1.0, INFIT mean-square). Divide that sum by the count of entries. This is the "Real Error variance" estimate. Then,

Real Reliability = True Variance / Observed Variance = (Observed Variance - Real Error Variance) / Observed Variance.

Separation, Strata and Reliability

The crucial elements in the computation of reliability are the "True" variance and the Error variance. These are squared distances and so difficulty to conceptualize directly. It is easier to think of their square-roots, the "True" standard deviation (TSD) and the root-mean-square standard error (RMSE).

SEPARATION coefficient is the ratio of the PERSON (or ITEM) TRUE S.D., the "true" standard deviation, to RMSE, the error standard deviation. It provides a ratio measure of separation in RMSE units, which is easier to interpret than the reliability correlation. This is analogous to the Fisher Discriminant Ratio. SEPARATION coefficient ² is the signal-to-noise ratio, the ratio of "true" variance to error variance.

RELIABILITY (separation index) is a separation reliability. The PERSON (or ITEM) reliability is equivalent to KR-20, Cronbach Alpha, and the Generalizability Coefficient. The relationship between SEPARATION coefficient and RELIABILITY (separation index) is

RELIABILITY = SEPARATION coefficient ²/(1+SEPARATION coefficient ²)

or SEPARATION coefficient = square-root(RELIABILITY/(1-RELIABILITY)).

Separation (if the outlying measures are accidental) or Strata (if the outlying measures represent true performances). These numbers are statistical abstractions, but there empirical meaning is indicated by locating the Separation or Strata levels in the observed distribution at (3 * "Observed S.D." / Separation) units apart, centered on the sample mean.

 Error RMSE True SD True Variance Observed Variance Signal- to-Noise Ratio Separation = True SD / RMSE Strata  = (4*Sep.+1) / 3 Reliability = True Variance / Observed Variance 1 0 0 1 0 0 0 0 1 1 1 2 1 1 1.67 0.5 1 2 4 5 2 2 3 0.8 1 3 9 10 3 3 4.33 0.9 1 4 16 17 4 4 5.67 0.94

There is more at www.rasch.org/rmt/rmt94n.htm and www.rasch.org/rmt/rmt163f.htm

Spearman-Brown Prophecy Formula for person "Test" reliability with different numbers of items (test lengths)

How many items (or persons) are required to produce the reliability I want with the sample of persons and the same type of items (or this test and the same type of persons)?

T = target number of items, RT = target person reliability

C = current number of items, RC = current person reliability

1. Predict number of items = T = C * RT * (1-RC) / ( (1-RT) * RC)

Example: the current test length is C = 10 items, and the current person reliability is RC = 0.3. We want a person reliability of RT = 0.8.

Target number of items is T = 10 * 0.8 * (1-0.3) / ( (1-0.8)* 0.3) = 94 items.

2. Predicted person reliability = RT = T * RC / ( C * (1-RC) + T * RC)

Example: we have a test of C = 11 items of person reliability RC = 0.5, what is the predicted reliability of a test of T = 17 items?

Predicted person reliability RT =  17 * 0.5 / ( 11 * (1-0.5) + 17 * 0.5) = 0.61

Prophecy Formula for person "Test" reliability with different observed standard deviations of the person measures/scores

assuming the average error variance is unchanged.

T = target observed standard deviation, RT = target person reliability

C = current observed standard deviation, RC = current person reliability

1. Predict person standard deviation = T = C * sqrt ( (1-RC) /  (1-RT) )

2, Predict person reliability = RT = 1 - ( (1-RC)*C² / T² )

Test-Retest Reliability

is the correlation between the person measures obtained from two administrations of the same test to the same persons. The expected value of the test-retest reliability is the person reliability of the first administration.

The "Smallest Detectable Difference" = "Smallest statistically significant difference in a person's measures" when a test is administered twice to a person under normal conditions = 1.96*(person standard deviation)*(test-retest reliability).

Population and Sample Standard Deviation, Reliability, Separation and Strata

Winsteps assumes that the N statistics being summarized are the entire population. It reports the population observed Standard Deviation = P.SD, "True" Standard Deviation = T.P.SD, Reliability = P.Rel, Separation = P.Sep and Strata = P.Strata, average error S.D.  = RMSE, where

P.Sep = √ ( P.Rel / (P.Rel + 1) )

RMSE = P.SD * √ ( 1 - P.Rel )

If the N statistics are a sample of the entire population, then the summary statistics are the sample observed Standard Deviation = S.SD,  "True" Standard Deviation = T.S.SD, Reliability = S.Rel, Separation = S.Sep and Strata = S.Strata, where

S.SD = P.SD * √ ( N / (N-1)) - the sample S.D. is bigger than the population S.D.

T.S.SD = √ ( S.SD**2 - RMSE**2)

S.Sep = T.S.SD / RMSE

S.Strata = (4 * S.Sep + 1) / 3

S.Rel = ( 1 + (N-1)*P.Rel ) / N

Help for Winsteps Rasch Measurement Software: www.winsteps.com. Author: John Michael Linacre

## Just released in June 2017: Winsteps 4.0 with Table 45 Cumulative Plot

### New and on-going: Masterchef Australia 2017: Rasch Measurement of Cooks with Table 45

 Forum Rasch Measurement Forum to discuss any Rasch-related topic

Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez
Winsteps Tutorials Facets Tutorials Rasch Discussion Groups

Coming Rasch-related Events
June 30 - July 29, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 31 - Aug. 3, 2017, Mon.-Thurs. Joint IMEKO TC1-TC7-TC13 Symposium 2017: Measurement Science challenges in Natural and Social Sciences, Rio de Janeiro, Brazil, imeko-tc7-rio.org.br
Aug. 7-9, 2017, Mon-Wed. In-person workshop and research coloquium: Effect size of family and school indexes in writing competence using TERCE data (C. Pardo, A. Atorressi, Winsteps), Bariloche Argentina. Carlos Pardo, Universidad Catòlica de Colombia
Aug. 7-9, 2017, Mon-Wed. PROMS 2017: Pacific Rim Objective Measurement Symposium, Sabah, Borneo, Malaysia, proms.promsociety.org/2017/
Aug. 10, 2017, Thurs. In-person Winsteps Training Workshop (M. Linacre, Winsteps), Sydney, Australia. www.winsteps.com/sydneyws.htm
Aug. 11 - Sept. 8, 2017, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Aug. 18-21, 2017, Fri.-Mon. IACAT 2017: International Association for Computerized Adaptive Testing, Niigata, Japan, iacat.org
Sept. 15-16, 2017, Fri.-Sat. IOMC 2017: International Outcome Measurement Conference, Chicago, jampress.org/iomc2017.htm
Oct. 13 - Nov. 10, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 5 - Feb. 2, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 10-16, 2018, Wed.-Tues. In-person workshop: Advanced Course in Rasch Measurement Theory and the application of RUMM2030, Perth, Australia (D. Andrich), Announcement
Jan. 17-19, 2018, Wed.-Fri. Rasch Conference: Seventh International Conference on Probabilistic Models for Measurement, Matilda Bay Club, Perth, Australia, Website
April 13-17, 2018, Fri.-Tues. AERA, New York, NY, www.aera.net
May 25 - June 22, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
<script type="text/javascript" src="http://www.rasch.org/events.txt"></script>

For more information, contact Winsteps.com by e-mail using the comment form above.
Our current URL is www.winsteps.com