In this Foreign Language reading test, examinees read two text passages. The first passage (chosen at random from many available passages) is heard by a judge. The examinee than moves to an intermediate waiting room. Then the examinee is heard by a different judge on a different, randomly selected, passage. Examinees are assigned to the next available judge at each stage. Rating is on a holistic 1-5 category scale.
Facets specifications and data (in file Pair.txt):
Title = Paired-judge Reading Test ; the report heading line Facets = 4 ; examinees, texts, raters, time-point Converge = .5, .1 ; converged when biggest score residual is .5 and biggest change is .1 logits Positive = 1, 3 ; examinees are able, raters are lenient Noncenter = 1 ; examinee measures relative to texts, raters, etc. Umean = 50, 5 ; user-friendly logit-rescaled measures: logit = 50 + 5 * logit Arrange = F,m ; tables output in Fit-ascending (F) and Measure-descending (m) order Usort = U ; sort residuals by unexpectedness Vertical = 1A, 3A, 2A ; Display examinees, raters, texts by name Model = ?,?,?,?,reading ; basic model: raters share the 5-point rating scale ?,?B,?,?B,reading ; text x time-point interactions: does practice help? * Rating scale = reading,R9,K ; 5 category scale, K=keep intermediate unobserved categories 1 = incoherent ; sounding out the words, or worse 5 = fluent ; as a native speaker * Labels= 1, Examinees 12, Stan 14, Amanda 23, Victor 30, Leslie 41, Don 52, Chin 62, Anne 73, David 82, Jacinta 90, Kallen 99, Ethne * 2, Passages 1, Buchan 2, Sapper 3, Fleming 4, Doyle 5, Montagu 6, Blyton * 3, Judges 1, Glass 2, Draper 3, Karnes 4, Borich 5, Chase 6, Gredler * 4, Order, A ; these are anchored at umean=50 because for interactions only 1, First, 50 2, Second, 50 * data= 12,1,1,1, 2 ; Stan reads Buchan to Glass first, and is rated 2 12,3,5,2, 3 ; Stan reads Fleming to Chase second, and is rated 3 14,6,4,1, 1 14,4,6,2, 2 23,1,4,1, 1 23,3,2,2, 3 30,1,5,1, 1 30,5,4,2, 2 41,3,5,1, 4 41,6,4,2, 3 52,4,6,1, 1 52,5,2,2, 3 62,4,3,1, 1 62,5,2,2, 3 73,4,3,1, 4 73,6,2,2, 3 82,1,6,1, 2 82,5,1,2, 5 90,3,1,1, 2 90,2,4,2, 4 99,6,3,1, 3 99,2,1,2, 5 |
In Table 7, the "Fair Average" reports a standard rating for a standard context:
Here, Kallen has a higher observed average score of 3.0, but lower "Fair Average" score of 2.14 because he read out easier passages to more lenient judges.