﻿ Diagnosing Misfit

Diagnosing Misfit

What do Infit Mean-square, Outfit Mean-square, Infit Zstd (z-standardized), Outfit Zstd (z-standardized) mean?

General rules:

Mean-squares show the size of the randomness, i.e., the amount of distortion of the measurement system. 1.0 are their expected values. Values less than 1.0 indicate observations are too predictable (redundancy, model overfit). Values greater than 1.0 indicate unpredictability (unmodeled noise, model underfit). Mean-squares usually average to 1.0, so if there are high values, there must also be low ones. Examine the high ones first, and temporarily remove them from the analysis if necessary, before investigating the low ones.

Zstd are t-tests of the hypotheses "do the data fit the model (perfectly)?". These are reported as z-scores, i.e., unit normal deviates. They show the improbability (significance). 0.0 are their expected values. Less than 0.0 indicate too predictable. More than 0.0 indicates lack of predictability. If mean-squares are acceptable, then Zstd can be ignored. They are truncated towards 0, so that 1.00 to 1.99 is reported as 1. So a value of 2 means 2.00 to 2.99, i.e., at least 2. See Score files for more exact values.

The Wilson-Hilferty cube root transformation converts the mean-square statistics to the normally-distributed z-standardized ones. For more information, please see Patel's "Handbook of the Normal Distribution".

Guidelines:

(a) Look for negative bi-serial correlations and large response residuals. Explain or eliminate these first.

(b) If Zstd is acceptable, usually <|2| or <|3|, then there may not be much need to look further.

(c) If mean-squares indicate only small departures from model-conditions, then the data are probably useful for measurement.

(d) If there are only small proportion of misfitting elements, including or omitting them will make no substantive difference. If in doubt, do analyses with and without them and compare results.

(e) If measurement improves without misfitting elements, then

(i) omit misfitting elements

(ii) do an analysis without them and produce an anchorfile=

(iii) edit the anchorfile= to reinstate misfitting elements.

(iv) do an analysis with the anchorfile.

The misfitting elements will now be placed in the measurement framework, but without degrading the measures of the other elements.

Anchored runs:

Anchor values may not exactly accord with the current data. To the extent that they don't, they fit statistics may be misleading. Anchor values that are too central for the current data tend to make the data appear to fit too well. Anchor values that are too extreme for the current data tend to make the data appear noisy.

Mean-square interpretation:

>2.0 Distorts or degrades the measurement system.

1.5 - 2.0 Unproductive for construction of measurement, but not degrading.

0.5 - 1.5 Productive for measurement.

<0.5 Less productive for measurement, but not degrading. May produce misleadingly good reliabilities and separations.

In general, mean-squares near 1.0 indicate little distortion of the measurement system, regardless of the Zstd value.

Evaluate high mean-squares before low ones, because the average mean-square is usually forced to be near 1.0.

Outfit mean-squares:  influenced by outliers. Usually easy to diagnose and remedy. Less threat to measurement.

Infit mean-squares: influenced by response patterns. Usually hard to diagnose and remedy. Greater threat to measurement.

 Diagnosing Misfit Classification INFIT OUTFIT Explanation Investigation Noisy Noisy Lack of convergence Loss of precision Anchoring Final values in Table 0 large? Many categories? Large logit range? Displacements reported? Hard Item Noisy Noisy Bad item Ambiguous or negative wording? Debatable or misleading options? Muted Muted Only answered by top people At end of test? Item Noisy Noisy Qualitatively different item Incompatible anchor value Different process or content? Anchor value incorrectly applied? ? Biased (DIF) item Stratify residuals by person group? Muted Curriculum interaction Are there alternative curricula? Muted ? Redundant item Similar items? One item answers another? Item correlated with other variable? Rating scale Noisy Noisy Extreme category overuse Poor category wording? Combine or omit categories? Wrong model for scale? Muted Muted Middle category overuse Person Noisy ? Processing error Clerical error Idiosyncratic person Scanner failure? Form markings misaligned? Qualitatively different person? High Person ? Noisy Careless Sleeping Rushing Unexpected wrong answers? Unexpected errors at start? Unexpected errors at end? Low Person ? Noisy Guessing Response set "Special" knowledge Unexpected right answers? Systematic response pattern? Content of unexpected answers? Muted ? Plodding Caution Did not reach end of test? Only answered easy items? Person/Judge Rating Noisy Noisy Extreme category overuse Extremism? Defiance? Muted Muted Middle category overuse Conservatism? Resistance? Judge Rating Apparent unanimity Collusion? INFIT: OUTFIT: Muted: Noisy: information-weighted mean-square, sensitive to irregular inlying patterns usual unweighted mean-square, sensitive to unexpected rare extremes unmodeled dependence, redundancy, error trends unexpected unrelated irregularities

Help for Facets Rasch Measurement Software: www.winsteps.com Author: John Michael Linacre.

Now in progress: The Batchelor - Australia - 2018: Rasch Measurement of Romance

 Forum Rasch Measurement Forum to discuss any Rasch-related topic

To receive News Emails about Winsteps and Facets by subscribing to the Winsteps.com email list,

I want to Subscribe: & click below
I want to Unsubscribe: & click below

The Winsteps.com email list is only used to email information about Winsteps, Facets and associated Rasch Measurement activities. Your email address is not shared with third-parties. Every email sent from the list includes the option to unsubscribe.

Questions, Suggestions? Want to update Winsteps or Facets? Please email Mike Linacre, author of Winsteps mike@winsteps.com

Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement with Raters and Rating Scales: Rasch Models for Rater-Mediated Assessments, George Engelhard, Jr. & Stefanie Wind Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez
Winsteps Tutorials Facets Tutorials Rasch Discussion Groups

Coming Winsteps & Facets Events
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Oct. 24 - 26, 2018,Wed.-Fri. Rasch workshop at Midwest Educational Research Association Annual Meeting, Cincinatti, Ohio (W. Boone, Winsteps), boonewjd@gmail.com
Jan. 25 - Feb. 22, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
April 4 - 8, 2019, Thur.-Mon. NCME annual meeting, Toronto, Canada.https://ncme.connectedcommunity.org/meetings/annual
April 5 - 9, 2019, Fri.-Tue. AERA annual meeting, Toronto, Canada.www.aera.net/Events-Meetings/Annual-Meeting
May 24 - June 21, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 28 - July 26, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 9 - Sept. 6, 2019, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 11 - Nov. 8, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 24 - Feb. 21, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 22 - June 19, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 26 - July 24, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 7 - Sept. 4, 2020, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith,Facets), www.statistics.com
Oct. 9 - Nov. 6, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 25 - July 23, 2021, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com

Our current URL is www.winsteps.com