Start with the Diagnosis menu ... Work your way down it.
In test construction, the guiding principle is "all items must be about the same thing, but then be as different as possible"! The central idea is that there is a latent variable which we are attempting to measure people on. The empirical definition of the latent variable is the content of the items. Essentially, we should be able to summarize on the items into a sentence which matches our intended definition of the latent variable. Latent variables can be very broad, e.g., "psychological state" or "educational achievement", or very narrow, e.g., "degree of paranoia" or "ability to do long division".
In other words, all items share something in common, but each item also brings in something that the others don't have.
Of course, this never happens perfectly. So what we need is:
(!) the CODES= statement contains the valid data codes in the data and nothing else.
(a) all items to point in the same direction, so that a higher rating (or "correct" answer) on the item indicates more of the latent variable. The first entry on the Diagnosis menu displays correlations. Items with negative correlations probably need their scoring reversed with IVALUE=.
(b) what the items share overwhelms what they don't share
(c) what the items don't all share, i.e., what is different about each of the items, is unique to each item or shared by only a few items.
What they all (or almost all) share, is usually thought of as the "test construct", the "major dimension", or the "Rasch dimension", or the "first factor in the data". This is what test validation studies focus on. Evaluating or confirming the nature of this construct.
What is unique to each item, or to clusters of only a few items, are "subdimensions", "secondary dimensions", "secondary contrasts", "misfit to the Rasch dimension", etc. We are concerned to evaluate: (i) are they a threat to scores/measures on the major dimension? (ii) do they manifest any useful information?
There are always as many contrasts in a test as there are items (less one). So how do we proceed?
(a) We want the first dimension to be much larger than all other dimensions, and for all items to have a large positive loading on it. This is essentially what the point-biserial correlation tells us in a rough way, and Rasch analysis tells us in a more exact way.
(b) We want so few items to load on each subdimension that we would not consider making that subdimension into a separate instrument. In practice, we would need at least 5 items to load heavily on a contrast, maybe more, before we would consider those items as a separate instrument. Then we crossplot and correlate scores or measures on the subdimension against scores on the rest of the test to see its impact.
(c) When a contrast has 2 items or less heavily loading on it, then it may be interesting, but it is only a wrinkle in this test. For instance, when we look at a two item contrast, we may say, "That is interesting, we could make a test of items like these!" But to make that test, we would need to write new items and collect more data. Its impact on this test is obviously minimal.
In reporting your results, you would want to:
(a) Describe, and statistically support, what most items share: the test construct.
(b) Identify, describe and statistically support, sub-dimensions big enough to be split off as separate tests. Then contrast scores/measures on those subdimensions with scores/measures on the rest of the test.
(c) Identify smaller sub-dimensions and speculate as to whether they could form the basis of useful new tests.
In all this, statistical techniques, like Rasch analysis and factor analysis, support your thinking, but do not do your thinking for you!
In what you are doing, I suggest that you choose the simplest analytical technique that enables you to tell your story, and certainly choose one that you understand!
Question: How do I know when to stop removing badly-fitting items and persons from my analysis?
Answer: You need to ask yourself a practical question: Why am I removing items and persons?
Is this your answer? "To improve the measurement of the persons!"
OK - then here is the strategy.
1. Estimate the person measures from the original analysis.
Remove whatever you see to be really, really bad.
2. Estimate the person measures again.
Cross-plot the two sets of person measures. Are there any noticeable changes that matter to you?
No. Then the really, really bad wasn't so bad after all. Keep everything. Stop here.
Yes. Now remove the really bad.
3. Estimate the person measures again.
Cross-plot the two sets of person measures (2. and 3.). Are there any noticeable changes that matter to you?
No. Then the really bad wasn't so bad after all. Analysis 2 is what you want. Stop here.
Yes. Now remove the somewhat bad.
4. Estimate the person measures again.
Cross-plot the two sets of person measures (3. and 4.). Are there any noticeable changes that matter to you?
No, then the somewhat bad wasn't so bad after all. Analysis 3 is what you want. Stop here.
Yes, ..... (and so on)
It is usual to discover that the recommended fit criteria are much too tight. That is because those criteria were formulated by statisticians concerned about model-fit. They were not formulated by practical people who are concerned about measurement quality.
There is an interesting parallel to this in industrial quality-control. Part of the reason for the huge advance in the quality of Japanese cars relative to American cars was because Japanese quality-control focused on the overall quality of the car (which is what the consumer wants), while American quality-control focused on the quality of individual components (which is what the automotive engineers want).
