Advice to novice analysts

Start with the Diagnosis menu ... Work your way down it.

 

In test construction, the guiding principle is "all items must be about the same thing, but then be as different as possible"! The central idea is that there is a latent variable which we are attempting to measure people on. The empirical definition of the latent variable is the content of the items. Essentially, we should be able to summarize on the items into a sentence which matches our intended definition of the latent variable. Latent variables can be very broad, e.g., "psychological state" or "educational achievement", or very narrow, e.g., "degree of paranoia" or "ability to do long division".

 

In other words, all items share something in common, but each item also brings in something that the others don't have.

 

Of course, this never happens perfectly. So what we need is:

 

(!) the CODES= statement contains the valid data codes in the data and nothing else.

 

(a) all items to point in the same direction, so that a higher rating (or "correct" answer) on the item indicates more of the latent variable. The first entry on the Diagnosis menu displays correlations. Items with negative correlations probably need their scoring reversed with IVALUE=.

 

(b) what the items share overwhelms what they don't share

 

(c) what the items don't all share, i.e., what is different about each of the items, is unique to each item or shared by only a few items.

 

What they all (or almost all) share, is usually thought of as the "test construct", the "major dimension", or the "Rasch dimension", or the "first factor in the data". This is what test validation studies focus on. Evaluating or confirming the nature of this construct.

 

What is unique to each item, or to clusters of only a few items, are "subdimensions", "secondary dimensions", "secondary contrasts", "misfit to the Rasch dimension", etc. We are concerned to evaluate: (i) are they a threat to scores/measures on the major dimension? (ii) do they manifest any useful information?

 

There are always as many contrasts in a test as there are items (less one). So how do we proceed?

 

(a) We want the first dimension to be much larger than all other dimensions, and for all items to have a large positive loading on it. This is essentially what the point-biserial correlation tells us in a rough way, and Rasch analysis tells us in a more exact way.

 

(b) We want so few items to load on each subdimension that we would not consider making that subdimension into a separate instrument. In practice, we would need at least 5 items to load heavily on a contrast, maybe more, before we would consider those items as a separate instrument. Then we crossplot and correlate scores or measures on the subdimension against scores on the rest of the test to see its impact.

 

(c) When a contrast has 2 items or less heavily loading on it, then it may be interesting, but it is only a wrinkle in this test. For instance, when we look at a two item contrast, we may say, "That is interesting, we could make a test of items like these!" But to make that test, we would need to write new items and collect more data. Its impact on this test is obviously minimal.

 

In reporting your results, you would want to:

 

(a) Describe, and statistically support, what most items share: the test construct.

 

(b) Identify, describe and statistically support, sub-dimensions big enough to be split off as separate tests. Then contrast scores/measures on those subdimensions with scores/measures on the rest of the test.

 

(c) Identify smaller sub-dimensions and speculate as to whether they could form the basis of useful new tests.

 

In all this, statistical techniques, like Rasch analysis and factor analysis, support your thinking, but do not do your thinking for you!

 

In what you are doing, I suggest that you choose the simplest analytical technique that enables you to tell your story, and certainly choose one that you understand!

 


 

Question: How do I know when to stop removing badly-fitting items and persons from my analysis?

 

Answer: You need to ask yourself a practical question: Why am I removing items and persons?

 

Is this your answer? "To improve the measurement of the persons!"

 

OK - then here is the strategy.

 

1. Estimate the person measures from the original analysis.

 

Remove whatever you see to be really, really bad.

 

2. Estimate the person measures again.

 

Cross-plot the two sets of person measures. Are there any noticeable changes that matter to you?

 

No. Then the really, really bad wasn't so bad after all. Keep everything. Stop here.

 

Yes. Now remove the really bad.

 

3. Estimate the person measures again.

 

Cross-plot the two sets of person measures (2. and 3.). Are there any noticeable changes that matter to you?

 

No. Then the really bad wasn't so bad after all. Analysis 2 is what you want. Stop here.

 

Yes. Now remove the somewhat bad.

 

4. Estimate the person measures again.

 

Cross-plot the two sets of person measures (3. and 4.). Are there any noticeable changes that matter to you?

 

No, then the somewhat bad wasn't so bad after all. Analysis 3 is what you want. Stop here.

 

Yes, ..... (and so on)

 

It is usual to discover that the recommended fit criteria are much too tight. That is because those criteria were formulated by statisticians concerned about model-fit. They were not formulated by practical people who are concerned about measurement quality.

 

There is an interesting parallel to this in industrial quality-control. Part of the reason for the huge advance in the quality of Japanese cars relative to American cars was because Japanese quality-control focused on the overall quality of the car (which is what the consumer wants), while American quality-control focused on the quality of individual components (which is what the automotive engineers want).


Help for Winsteps Rasch Measurement Software: www.winsteps.com. Author: John Michael Linacre

The Languages of Love: draw a map of yours!

For more information, contact info@winsteps.com or use the Contact Form
 

Facets Rasch measurement software. Buy for $149. & site licenses. Freeware student/evaluation download
Winsteps Rasch measurement software. Buy for $149. & site licenses. Freeware student/evaluation download

State-of-the-art : single-user and site licenses : free student/evaluation versions : download immediately : instructional PDFs : user forum : assistance by email : bugs fixed fast : free update eligibility : backwards compatible : money back if not satisfied
 
Rasch, Winsteps, Facets online Tutorials

 

Forum Rasch Measurement Forum to discuss any Rasch-related topic

Click here to add your email address to the Winsteps and Facets email list for notifications.

Click here to ask a question or make a suggestion about Winsteps and Facets software.

Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement with Raters and Rating Scales: Rasch Models for Rater-Mediated Assessments, George Engelhard, Jr. & Stefanie Wind Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez
Winsteps Tutorials Facets Tutorials Rasch Discussion Groups

 


 

 
Coming Winsteps & Facets Events
May 22 - 24, 2018, Tues.-Thur. EALTA 2018 pre-conference workshop (Introduction to Rasch measurement using WINSTEPS and FACETS, Thomas Eckes & Frank Weiss-Motz), https://ealta2018.testdaf.de
May 25 - June 22, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 27 - 29, 2018, Wed.-Fri. Measurement at the Crossroads: History, philosophy and sociology of measurement, Paris, France., https://measurement2018.sciencesconf.org
June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 25 - July 27, 2018, Wed.-Fri. Pacific-Rim Objective Measurement Symposium (PROMS), (Preconference workshops July 23-24, 2018) Fudan University, Shanghai, China "Applying Rasch Measurement in Language Assessment and across the Human Sciences" www.promsociety.org
Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com

 

 

Our current URL is www.winsteps.com

Winsteps® is a registered trademark
 


 
John "Mike" L.'s Wellness Report: I'm 72, take no medications and, March 2018, my doctor is annoyed with me - I'm too healthy!
According to Wikipedia, the human body requires about 30 minerals, maybe more. There are 60 naturally-occurring minerals in the liquid Mineral Supplement which I take daily.