Hi, it's been a while since I've done any Rasch analyses or posted on here. I have now had to analyse a data file and am trying to get my head around as to why I got so many negative point biserial correlations.
The data are judgments of 6 experts on how demanding certain test items are on a scale of 1 (less demanding) to 5 (more demanding). They all rated all the items.
I analysed the data in Facets though it is only a two-facet design.
The fit stats, reliability etc. are generally fine, and point biserial correlations for the judges are fine. The rating scale seems to have functioned well too.
However, there are lots of negative correlations for the items. I don't think it is the reversal problem, but am not sure what else it could be.
I attach the output file so you can see for yourself.

Thanks very much for your suggestions!
Milja

Yes, Milja, this is a surprising report. The judges and the rating scale are functioning well. The surprise is only in the items. The problem may be item MCQ2. This has huge misfit (mean-squares 2.7+).

1. Please rerun the analysis without this item (comment it out).

2. Please try Ptbiserial = measure. This will report both the point-measure correlation and its expectation. Then we can see whether the correlations are near to their expectations.

3. This is a small data set, so it could be analyzed with Ministep, the free version of Winsteps. Does that give the same report?

Hi Mike,
Thanks for the speedy response.
I attach the facets analyis with MCQ excluded and with expected pt biserial. Correlations are  a bit better without MCQ2 but there are still several negative ones...
Does this suggest anything to you?

I will do the Winsteps analysis tomorrow from work.

Best regards,
Milja

Thank you, Milja. The output makes sense, surprisingly!

The problem is that the 3 of the judges have the same measure. 2 more judges also have the same measure, close to the 3. Only one judge is far from the other judges. Consequently, for the item point-correlations, the one outlying judge is very influential.

If outlying judge 6 to rates lower than the average of the other judges, then the item point-correlation will be positive. If Judge 6 rates higher than the average of the other judges, then the item point-correlation will be negative. We see that judge 6 has an outfit mean-square of 1.47, so gives unexpected ratings more often than expected.

How about trying a "judge style" model?
Model = #,?,DEMANDS,1
This will show how each judge used the rating scale.

Thanks for this, Mike.
Please see attached output.
Not sure if this is important, but in the specification I have positive=2 (items), whereas judges should be negative.
I think here judge 6 is rating lower than average actually - so you would expect positive correlations then? Or am I misunderstanding something?

In any case, do you think this data set is too small to be analysed in this way without getting these sorts of anomalies? It is maybe unexpected to see that 3 judges had exactly the same measure (especially as they were not standardised in using this rating scale). So it is perhaps just by chance.

Thanks for all your help!

Your # analysis is informative, Milja.

Facets reports correlations so that the expected correlations are always positive. We do not need to remember whether the underlying facet is positive or negative.

Looking at Table 8 for the raters. Please look at the "Average Measure" column for each judge.  Judges 3, 5, 6 all have disordered average measures (average measure does not increase with rating scale category). This is probably partly due to the small sample size. Based on these data, only judge 2 is a reliable judge. The other 5 judges all have idiosyncrasies.