Mike,

If I recall correctly, you mentioned that a good way to determine if PCM should be used or whether items can all share the same rating scale, is to group the items according to what you think best using PCM, then pick an item from each group and compare the ICC's. Items from those curves that don't differ greatly could likley all be grouped together. I've attached a plot. One curve represents 6 items and the other 4. Using PCM I picked one item from each group. The low rated and mid rated respondents seem to perform slightly differently while high rated respondents seem to rate identical. The problem I have is making a judgment about whether the differences are significant or not. There were about 560 respondents.

Is there a way to determine the differences between the two curves using perhaps the data from the plots which are also in the attachment? If so, what criteria would I be using to support the visual?

Thanks as always.

Uve, if everyone responds to every item, then the different curvatures of the PCM ICCs will have a slight caterpillar effect on the person measures on the latent variable, but will not alter the rank order of the persons.

In general, we would be concerned if  the differences in the curvature of the ICCs caused a change in the rank-order of the item difficulties that impacted construct validity. We would need to ask ourselves, "Which do we trust more: the current data or the construct theory?" Physical scientists would reply "The construct theory, unless the data clearly and convincingly contradict it. In which case, the construct theory needs to be revised."

Suggestion: analyze the items using group-level rating scales as far as possible. Then simulate a dataset of the same size as the current dataset. Analyze the simulated dataset with PCM. How does the variation in ICCs compare with a PCM analysis of the original data?

Mike,

I did as you said. Item 1 of my survey had three choices and the other 13 items had 4. Though category 2 occupied an exteremely vary narrow range for the latter group of 13 items and I did collapse it with category 1, I still ran the analysis giving item 1 its own group and 2-14 the second group. I then created a simulated file and ran that at the lowest PCM level I felt was reasonable, 4 groups in all. I then picked 1 item from each of those groups from the simulated file and compared their order to the same 4 items from the first file. Though I did notice curve shape changes, the items seemed to be in the same order when choosing the Absolute X-axis option.

Since the order seems to be the same, I'm assuming that the original 2 group option is viable and that the 4 groups is not needed. However, had the 4 group PCM showed a different order, then I'm assuming it would have been the better choice. Would that be correct?

Uve, "had the 4 group PCM showed a different order, then I'm assuming it would have been the better choice. Would that be correct?"

The quantitative guys would say, "Of course!" The qualitative guys would say, "But that's only one piece of the evidence."

There's an interesting lesson in the recent Kaggle-Grockit competition (predicting the correctness of students responses to test items). The winner of the competition reported that he looked at the meaning behind his predictions. The runner-up reported that he merely looked at the numbers and ignored their meaning. This is the same challenge that the data-miners are facing: http://gizmodo.com/5906204/the-problem-with-big-data-is-that-nobody-understands-it

Thanks Mike.

Always so much more to think about  

I wonder if PCM and RSM would be analogous to viewing “A Sunday Afternoon on the Island of La Grande Jatte”  by Georges Seurat ( http://www.most-famous-paintings.org/A-Sunday-Afternoon-on-the-Island-of-La-Grande-Jatte---2-large.html ) a mere inch away, then 5 feet away. The meaning of the painting would certainly change.