Mike,

I am continuing my comparison of the advantages of the Rasch model as compared to CTT using SPSS with the same survey data in my previous post. This time I ran 4 separate reliablity analyses for two comparisons:

1) Comparison of the control group versus the treatment group on all 17 items
2) Comparison of items 1-11 versus 12-17 on all persons

SPSS Cronbach's Alpha:

1) .81  .82
2) .83  .79

Winsteps Cronbach:

1)  .97   .81
2) 1.00  1.00

Winsteps Model Person Reliability, All Persons

1)  .85   .66
2)  .81   .81

Again, I am wondering why there is such a great difference between the two. I realize that model reliability is a different calculation and so may not be directly comparable, but others are confusing me. Thanks for taking the time as always to look this over.

Thank you for these computations, Uve.

Based on their theories, we expect Cronbach Alpha to be greater than Rasch reliability, see
http://www.rasch.org/rmt/rmt113l.htm
- a situation many analysts have noticed in practice.

As an independent check on Cronbach Alpha (=KR-20 for dichotomies), I analyzed Guilford's Table 17.2. Here it is:

Title = "Guilford Table 17.2. His KR-20 = 0.81"
ni=8
item1=1
name1=1
&END
END LABELS
00000000
10000000
10100000
11001000
01010010
11101010
11111100
11111100
11110101
11111111

He reports KR-20 = 0.81. Winsteps reports Cronbach Alpha = 0.82. The difference is probably computational precision and rounding error.

Uve, what does SPSS report?

Mike,

SPSS reports .815

To be honest, for my purposes I would not be that concerned over a .01 or .02 difference. What concerned me more was the that Winsteps reported a perfect correlation of 1.00 for items 1-11 as well as items 12-17 while SPSS reported these as .83 and .79 respectively, which is a significant difference. Even the overall Cronbach in SPSS of .88, which I didn't mention before,  is significantly lower than the Winsteps Cronbach of .93. I can't help but think I've done something serioulsy wrong here or am interpreting a technique incorrectly.

Uve, good. It looks like SPSS and Winsteps agree about the basic computation.
Here are some areas where there can be differences:
1. Missing data.
2. Polytomous data.
3. Weighted data.
4. Rescored/recounted data.
Do any of these apply to your situation?

Yes: 1, 2 and 4. The data was attached in my previous SPSS and Table 23 comparison.

Several students did not responde to all data.

There are 4 options for each item.

None of the options are weighted differently

I received the initial file in SPSS with items 12-17 scored 1-4 for some reason while items 1-11 were scored 0-3. I simply changed all student responses to these last 6 items as opposed to rescoring them, which would probably have been easier.

Ouch, Uve! Both polytomous and missing data! So we need to isolate the difference.

1) Please choose a subset of polytomous dataline with no missing data. How do Winsteps and SPSS compare? They should agree.

2) Please choose a dichotomous dataset, such as exam1.txt. Make some observations missing. How do Winsteps and SPSS compare? They may differ. Winsteps skips missing observations in its computation, but SPSS may use case-wise deletion.

Thanks Mike. That must be it. I chose a sub-sample of 21 students who responded to all 17 items and compared both Winsteps and SPSS Cronbach. They were identical at .78.

I then took a closer look at the original SPSS output and found this message:

"Listwise deletion based on all variables in the procedure."

There were a total of 121 respondents, but 3 chose no options to any of the items. I deleted these in the Winsteps file but kept them in the SPSS file. SPSS deleted them as well along with 18 others for a total of 21. So I guess it was working off of 100 valid cases.

There were also a few students who responded to only two or three items and I deliberated for quite some time as to whether I should remove them from the Winsteps file but decided against it.

So with a listwise deletion process, are both SPSS and Winsteps treating missing data in the same manner?

Thanks again as always!                    

Uve, listwise deletion drops the data record if even one observation is missing. Winsteps does not do this. Winsteps computes the Cronbach variance terms using all the non-missing observations. For example, on a CAT test, listwise deletion would omit every person. Winsteps keeps every person.

So, if SPSS does automatic listwise deletion, and we deliberately delete records with missing data from a Winsteps analysis (using IDELETE= etc.), then the two computations should be the same.

Mike,

Continuing the discussion of missing data but for a different purpose, my data had about 4% missing responses from various respondents to various items. With 118 respondents and 17 items with 4 Likert options each, can one use a percentage as a rough guide for when it might be best to switch from Mantel-Heanszel probabilities to Welch t-tests for DIF?

Also, I understand that DIF is primarily intended to pick up on individual item idiosyncrasies and one can have DIF without multi-dimensionality. But is the reverse true?

Uve, MH is based on cross-tabs with one cross-tab for each ability stratum. MH is implemented in Winsteps using "thin" slicing (one stratum for each raw-score level).

Look at Winsteps Table 20.2, this will show you how many respondents there are at each score-level (FREQUENCY). Generally we would need at least 10 respondents in each cell of the cross-tab, so that would be at least 40 respondents in each stratum. If many stratum have fewer than 40, then increase MHSLICE= to encompass two or more ability strata.

Multidimensional items (e.g. a geography item in an arithmetic test) without DIF would indicate that the ability distribution on geography matches the ability distribution on arithmetic for both the focal and reference groups.