Diagnosing Misfit

What do Infit Mean-square, Outfit Mean-square, Infit Zstd (z-standardized), Outfit Zstd (z-standardized) mean?

 

General rules:

Mean-squares show the size of the randomness, i.e., the amount of distortion of the measurement system. 1.0 are their expected values. Values less than 1.0 indicate observations are too predictable (redundancy, model overfit). Values greater than 1.0 indicate unpredictability (unmodeled noise, model underfit). Mean-squares usually average to 1.0, so if there are high values, there must also be low ones. Examine the high ones first, and temporarily remove them from the analysis if necessary, before investigating the low ones.

 

Zstd are t-tests of the hypotheses "do the data fit the model (perfectly)?". These are reported as z-scores, i.e., unit normal deviates. They show the improbability (significance). 0.0 are their expected values. Less than 0.0 indicate too predictable. More than 0.0 indicates lack of predictability. If mean-squares are acceptable, then Zstd can be ignored. They are truncated towards 0, so that 1.00 to 1.99 is reported as 1. So a value of 2 means 2.00 to 2.99, i.e., at least 2. See Score files for more exact values.

 

The Wilson-Hilferty cube root transformation converts the mean-square statistics to the normally-distributed z-standardized ones. For more information, please see Patel's "Handbook of the Normal Distribution".

 

Guidelines:

(a) Look for negative bi-serial correlations and large response residuals. Explain or eliminate these first.

(b) If Zstd is acceptable, usually <|2| or <|3|, then there may not be much need to look further.

(c) If mean-squares indicate only small departures from model-conditions, then the data are probably useful for measurement.

(d) If there are only small proportion of misfitting elements, including or omitting them will make no substantive difference. If in doubt, do analyses with and without them and compare results.

(e) If measurement improves without misfitting elements, then

(i) omit misfitting elements

(ii) do an analysis without them and produce an anchorfile=

(iii) edit the anchorfile= to reinstate misfitting elements.

(iv) do an analysis with the anchorfile.

The misfitting elements will now be placed in the measurement framework, but without degrading the measures of the other elements.

 

Anchored runs:

Anchor values may not exactly accord with the current data. To the extent that they don't, they fit statistics may be misleading. Anchor values that are too central for the current data tend to make the data appear to fit too well. Anchor values that are too extreme for the current data tend to make the data appear noisy.

 

Mean-square interpretation:

>2.0 Distorts or degrades the measurement system.

1.5 - 2.0 Unproductive for construction of measurement, but not degrading.

0.5 - 1.5 Productive for measurement.

<0.5 Less productive for measurement, but not degrading. May produce misleadingly good reliabilities and separations.

 

In general, mean-squares near 1.0 indicate little distortion of the measurement system, regardless of the Zstd value.

 

Evaluate high mean-squares before low ones, because the average mean-square is usually forced to be near 1.0.

 

Outfit mean-squares:  influenced by outliers. Usually easy to diagnose and remedy. Less threat to measurement.

Infit mean-squares: influenced by response patterns. Usually hard to diagnose and remedy. Greater threat to measurement.

 

Diagnosing Misfit

Classification

INFIT

OUTFIT

Explanation

Investigation

 

Noisy

Noisy

Lack of convergence

Loss of precision

Anchoring

Final values in Table 0 large?

Many categories? Large logit range?

Displacements reported?

Hard Item

Noisy

Noisy

Bad item

Ambiguous or negative wording?

Debatable or misleading options?

Muted

Muted

Only answered by top people

At end of test?

Item

Noisy

Noisy

Qualitatively different item

Incompatible anchor value

Different process or content?

Anchor value incorrectly applied?

?

Biased (DIF) item

Stratify residuals by person group?

Muted

Curriculum interaction

Are there alternative curricula?

Muted

?

Redundant item

Similar items?

One item answers another?

Item correlated with other variable?

Rating scale

Noisy

Noisy

Extreme category overuse

Poor category wording?

Combine or omit categories?

Wrong model for scale?

Muted

Muted

Middle category overuse

Person

Noisy

?

Processing error

Clerical error

Idiosyncratic person

Scanner failure?

Form markings misaligned?

Qualitatively different person?

High Person

?

Noisy

Careless

Sleeping

Rushing

Unexpected wrong answers?

Unexpected errors at start?

Unexpected errors at end?

Low Person

?

Noisy

Guessing

Response set

"Special" knowledge

Unexpected right answers?

Systematic response pattern?

Content of unexpected answers?

Muted

?

Plodding

Caution

Did not reach end of test?

Only answered easy items?

Person/Judge Rating

Noisy

Noisy

Extreme category overuse

Extremism? Defiance?

Muted

Muted

Middle category overuse

Conservatism? Resistance?

Judge Rating

Apparent unanimity

Collusion?

INFIT:

OUTFIT:

Muted:

Noisy:

information-weighted mean-square, sensitive to irregular inlying patterns

usual unweighted mean-square, sensitive to unexpected rare extremes

unmodeled dependence, redundancy, error trends

unexpected unrelated irregularities


Help for Facets Rasch Measurement Software: www.winsteps.com Author: John Michael Linacre.
 

Now in progress: The Batchelor - Australia - 2018: Rasch Measurement of Romance

Facets Rasch measurement software. Buy for $149. & site licenses. Freeware student/evaluation download
Winsteps Rasch measurement software. Buy for $149. & site licenses. Freeware student/evaluation download

State-of-the-art : single-user and site licenses : free student/evaluation versions : download immediately : instructional PDFs : user forum : assistance by email : bugs fixed fast : free update eligibility : backwards compatible : money back if not satisfied
 
Rasch, Winsteps, Facets online Tutorials

Forum Rasch Measurement Forum to discuss any Rasch-related topic

To receive News Emails about Winsteps and Facets by subscribing to the Winsteps.com email list,
enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Winsteps.com
The Winsteps.com email list is only used to email information about Winsteps, Facets and associated Rasch Measurement activities. Your email address is not shared with third-parties. Every email sent from the list includes the option to unsubscribe.

Questions, Suggestions? Want to update Winsteps or Facets? Please email Mike Linacre, author of Winsteps mike@winsteps.com

Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement with Raters and Rating Scales: Rasch Models for Rater-Mediated Assessments, George Engelhard, Jr. & Stefanie Wind Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez
Winsteps Tutorials Facets Tutorials Rasch Discussion Groups

 

Coming Winsteps & Facets Events
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Oct. 24 - 26, 2018,Wed.-Fri. Rasch workshop at Midwest Educational Research Association Annual Meeting, Cincinatti, Ohio (W. Boone, Winsteps), boonewjd@gmail.com
Jan. 25 - Feb. 22, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
April 4 - 8, 2019, Thur.-Mon. NCME annual meeting, Toronto, Canada.https://ncme.connectedcommunity.org/meetings/annual
April 5 - 9, 2019, Fri.-Tue. AERA annual meeting, Toronto, Canada.www.aera.net/Events-Meetings/Annual-Meeting
May 24 - June 21, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 28 - July 26, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 9 - Sept. 6, 2019, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 11 - Nov. 8, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 24 - Feb. 21, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 22 - June 19, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 26 - July 24, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 7 - Sept. 4, 2020, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith,Facets), www.statistics.com
Oct. 9 - Nov. 6, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 25 - July 23, 2021, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com

 

Our current URL is www.winsteps.com

Winsteps® is a registered trademark