﻿ Diagnosing Misfit

# Diagnosing Misfit

What do Infit Mean-square, Outfit Mean-square, Infit Zstd (z-standardized), Outfit Zstd (z-standardized) mean?

General rules:

Mean-squares show the size of the randomness, i.e., the amount of distortion of the measurement system. 1.0 are their expected values. Values less than 1.0 indicate observations are too predictable (redundancy, model overfit). Values greater than 1.0 indicate unpredictability (unmodeled noise, model underfit). Mean-squares usually average to 1.0, so if there are high values, there must also be low ones. Examine the high ones first, and temporarily remove them from the analysis if necessary, before investigating the low ones.

Zstd are t-tests of the hypotheses "do the data fit the model (perfectly)?". These are reported as z-scores, i.e., unit normal deviates. They show the improbability (significance). 0.0 are their expected values. Less than 0.0 indicate too predictable. More than 0.0 indicates lack of predictability. If mean-squares are acceptable, then Zstd can be ignored. They are truncated towards 0, so that 1.00 to 1.99 is reported as 1. So a value of 2 means 2.00 to 2.99, i.e., at least 2. See Score files for more exact values.

The Wilson-Hilferty cube root transformation converts the mean-square statistics to the normally-distributed z-standardized ones. For more information, please see Patel's "Handbook of the Normal Distribution".

Guidelines:

(a) Look for negative bi-serial correlations and large response residuals. Explain or eliminate these first.

(b) If Zstd is acceptable, usually <|2| or <|3|, then there may not be much need to look further.

(c) If mean-squares indicate only small departures from model-conditions, then the data are probably useful for measurement.

(d) If there are only small proportion of misfitting elements, including or omitting them will make no substantive difference. If in doubt, do analyses with and without them and compare results.

(e) If measurement improves without misfitting elements, then

(i) omit misfitting elements

(ii) do an analysis without them and produce an anchorfile=

(iii) edit the anchorfile= to reinstate misfitting elements.

(iv) do an analysis with the anchorfile.

The misfitting elements will now be placed in the measurement framework, but without degrading the measures of the other elements.

Anchored runs:

Anchor values may not exactly accord with the current data. To the extent that they don't, they fit statistics may be misleading. Anchor values that are too central for the current data tend to make the data appear to fit too well. Anchor values that are too extreme for the current data tend to make the data appear noisy.

Mean-square interpretation:

>2.0 Distorts or degrades the measurement system.

1.5 - 2.0 Unproductive for construction of measurement, but not degrading.

0.5 - 1.5 Productive for measurement.

<0.5 Less productive for measurement, but not degrading. May produce misleadingly good reliabilities and separations.

In general, mean-squares near 1.0 indicate little distortion of the measurement system, regardless of the Zstd value.

Evaluate high mean-squares before low ones, because the average mean-square is usually forced to be near 1.0.

Outfit mean-squares:  influenced by outliers. Usually easy to diagnose and remedy. Less threat to measurement.

Infit mean-squares: influenced by response patterns. Usually hard to diagnose and remedy. Greater threat to measurement.

 Diagnosing Misfit Classification INFIT OUTFIT Explanation Investigation Noisy Noisy Lack of convergence Loss of precision Anchoring Final values in Table 0 large? Many categories? Large logit range? Displacements reported? Hard Item Noisy Noisy Bad item Ambiguous or negative wording? Debatable or misleading options? Muted Muted Only answered by top people At end of test? Item Noisy Noisy Qualitatively different item Incompatible anchor value Different process or content? Anchor value incorrectly applied? ? Biased (DIF) item Stratify residuals by person group? Muted Curriculum interaction Are there alternative curricula? Muted ? Redundant item Similar items? One item answers another? Item correlated with other variable? Rating scale Noisy Noisy Extreme category overuse Poor category wording? Combine or omit categories? Wrong model for scale? Muted Muted Middle category overuse Person Noisy ? Processing error Clerical error Idiosyncratic person Scanner failure? Form markings misaligned? Qualitatively different person? High Person ? Noisy Careless Sleeping Rushing Unexpected wrong answers? Unexpected errors at start? Unexpected errors at end? Low Person ? Noisy Guessing Response set "Special" knowledge Unexpected right answers? Systematic response pattern? Content of unexpected answers? Muted ? Plodding Caution Did not reach end of test? Only answered easy items? Person/Judge Rating Noisy Noisy Extreme category overuse Extremism? Defiance? Muted Muted Middle category overuse Conservatism? Resistance? Judge Rating Apparent unanimity Collusion? INFIT: OUTFIT: Muted: Noisy: information-weighted mean-square, sensitive to irregular inlying patterns usual unweighted mean-square, sensitive to unexpected rare extremes unmodeled dependence, redundancy, error trends unexpected unrelated irregularities

Help for Facets Rasch Measurement Software: www.winsteps.com Author: John Michael Linacre.

 Forum Rasch Measurement Forum to discuss any Rasch-related topic

Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement with Raters and Rating Scales: Rasch Models for Rater-Mediated Assessments, George Engelhard, Jr. & Stefanie Wind Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez
Winsteps Tutorials Facets Tutorials Rasch Discussion Groups

Coming Rasch-related Events
Jan. 5 - Feb. 2, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 10-16, 2018, Wed.-Tues. In-person workshop: Advanced Course in Rasch Measurement Theory and the application of RUMM2030, Perth, Australia (D. Andrich), Announcement
Jan. 17-19, 2018, Wed.-Fri. Rasch Conference: Seventh International Conference on Probabilistic Models for Measurement, Matilda Bay Club, Perth, Australia, Website
Jan. 22-24, 2018, Mon-Wed. In-person workshop: Rasch Measurement for Everybody en español (A. Tristan, Winsteps), San Luis Potosi, Mexico. www.ieia.com.mx
April 10-12, 2018, Tues.-Thurs. Rasch Conference: IOMW, New York, NY, www.iomw.org
April 13-17, 2018, Fri.-Tues. AERA, New York, NY, www.aera.net
May 22 - 24, 2018, Tues.-Thur. EALTA 2018 pre-conference workshop (Introduction to Rasch measurement using WINSTEPS and FACETS, Thomas Eckes & Frank Weiss-Motz), https://ealta2018.testdaf.de
May 25 - June 22, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 27 - 29, 2018, Wed.-Fri. Measurement at the Crossroads: History, philosophy and sociology of measurement, Paris, France., https://measurement2018.sciencesconf.org
June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 25 - July 27, 2018, Wed.-Fri. Pacific-Rim Objective Measurement Symposium (PROMS), (Preconference workshops July 23-24, 2018) Fudan University, Shanghai, China "Applying Rasch Measurement in Language Assessment and across the Human Sciences" www.promsociety.org
Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Sept. 3 - 6, 2018, Mon.-Thurs. IMEKO World Congress, Belfast, Northern Ireland www.imeko2018.org
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com

Our current URL is www.winsteps.com