Fair average

A "Fair Average" for each element of the target facet is computed as though every observation of that element in the data was for an average element of every other facet. Then these artificial observations are averaged by the number of observations of the target element. This is "fair" in the sense that every element of the target facet encounters elements of the same logit values in the other facets. The Fair Average is computed using an average rater, essay, etc. Since the mathematics is non-linear (logistic), a rating by an average rater is not the same as the average of the ratings by all the raters, but is usually close to it.

 

We can check this by outputting an Anchorfile=. Keep everything anchored. Then, for every facet except one (the target), change the anchor values of the elements to the average logit value of the elements in that facet.  Then the expected average for each element in the target facet should be the same as the fair average.

 

Meaning of the Fair Average:

 

The "Fair Average" is what the raters would have had if they had all rated the same "average" elements under the same "average" conditions. For instance, suppose one rater rated all the easy tasks (= high observed average rating)  but another rater rated all the hard tasks (= low observed average rating). Then the "Fair Average" says "What if both those raters had rated the same average task, what would the rating have been?". We can then use the Fair Average to compare the severity/leniency of the raters as though they had rated under the same conditions.

 

Example: What's the relationship between the raw score and Fair Averages?

They are both in the original rating-scale metric. The raw score on an item is the original observation. The Fair Average is the original observation adjusted for its context. Suppose that my performance receives a rating of 3 from a lenient rater. My Fair Average is 2.5. Your performance receives a rating of 3 from a severe rater. Your Fair Average is 3.5. Comparing your "3" with my "3" is unfair, because you had a severe rater (who generally gives low ratings) and I had a lenient rater (who generally gives high ratings). After adjusting for rater severity/leniency, our "Fair Average" ratings are 2.5 and 3.5. These give a fair comparison.

 

The "fair average" transforms the Rasch measure back into an expected average raw response value. This value is in a standardized environment in which all other elements interacting with this element have a zero measure or the mean measure of all elements in their facet. This is "fair" to all elements in the facet, e.g., this adjusts raw ratings for severe and lenient raters. This enables a "fair" comparison to be made in the raw score metric, in the same way that the measure does on the linear latent variable. Fair-M uses the facet element means as the baseline. Fair-Z uses the facet local origins (zero points) as the baseline. These are set by Fair average=.

 

The original purpose of Facets (in 1986) was to construct software that would automatically adjust for differences in rater severity/leniency. So this has been done for your data. No adjustment or trimming of usefully-fitting raters is necessary, regardless of their severity/leniency.

 

But Facets does assume that the average leniency of the raters is at the required standard of severity/leniency (usually by centering the rater facet at zero logits). If you see, from evidence external to the data, that the average leniency of the of the raters is too high or too low, then please

1. include an additional "adjustment" facet in your analysis with a ? in Models=

2. this facet has one element.

3. Anchor this element at the adjustment value

4. Include the element in all the observations in the dataset by using dvalues=

5. perform an analysis of all the data including the adjustment element The rater leniencies are now correct, but the fair averages are not correct for the adjusted leniencies.

6. output an anchorfile

7. replace the ? in the model specification with an X

8. analyze the anchorfile with the dataset

9. there will be displacements of the size of the adjustment value. The fair averages are correct for the adjusted leniencies.

 

This procedure will require several attempts before it produces the correct results.

 

Standard Error of the Fair Average:

Probably the easiest way to obtain statistically rigorous estimates of the S.E.s of the Fair Averages is to use Facets to simulate 20 or more datasets based on the current dataset. Then the S.D.s of the 20 or more replications of each Fair Average is the S.E. of the Fair Average.

 

We can approximate the S.E. of a Fair Average by using two elements from the Facet as reported in Table 7.

facet with similar characteristics (same number of ratings, similar rating situations) but a different measure.

S.E. of Fair score for element 1 =  S.E. of Measure for element 1 * (Fair score for element 1 - Fair score for element 2) / (Measure for element 1 - Measure for element 2)

Standard errors are always written as positive numbers.

 

Example: Anne and Chris were rated the same number of times in the same type of rating situations:

Anne: Fair average = 4.64  measure = -.07  S.E. of measure = .17

Chris: Fair average = 4.23  measure = -.25

S.E. of Anne's Fair average = .17 * (4.64 - 4.23) / (-.07 - -0.25) = 0.39 score points.

 

Calculation of the Fair Average Score

The observed average score is the average rating received by the element. The logit measure is the linear measure implied by the observations. This is adjusted for the measures of whatever other elements of other facets participated in producing the observed data. It is often useful to transform these measures back into the original raw score metric to communicate their substantive meaning. Fair Average does this. It is the observed average adjusted for the measures of the other elements encountered. It is the observed average that would have been received if all the measures of the other elements had been located at the average measure of the elements in each of their facets.

 

A basic many-facet Rasch model for observation Xnmij is:

 

 log ( Pnmijk / Pnmij(k-1)) = Bn - Am - Di - Cj - Fk

 

where

Bn is the ability of person n, e.g., examinee: Mary,

Am is the challenge of task m, e.g., an essay: "My day at the zoo".

Di is the difficulty of item i, e.g., punctuation,

Cj is the severity of judge j, e.g., the grader: Dr. Smith,

Fk is the barrier to being observed in category k relative to category k-1,

where k=0 to t, and F0=0.

 

To compute the fair average for person n (or task m, item i, judge j), set all element parameters except Bn (or Am, Di, Cj) to their mean (or zero) values. Thus, the model underlying a fair rating, when Fair=Mean, is:

 

 log ( Pnmijk / Pnmij(k-1)) = Bn - Amean - Dmean - Cmean - Fk

 

or, when Fair=Zero, it becomes:

 

 log ( Pnmijk / Pnmij(k-1)) = Bn - Fk

 

and the Fair average is sum(k Pnmijk) across categories k=0 to t.

 

Example: Students are rated on 6 items. I want a fair-average for each student on each item.

1. Perform the analysis of all the data.

2. Output an Anchorfile=

3. Unanchor the students (remove the ,A for students in Labels=)

4. Analyze the anchorfile one item at a time by commenting out all items except 1 in Labels=

5. The reported measures and fair-averages for the students will be the fair averages for each item.

6. To assemble these fair-averages, output a Scorefile= from each one-item analysis to, say, Excel, selecting student identification and fair average fields.

 


 

Verifying the Fair Average

 

Analyze your data in Facets. Choose a Facet and an element whose Fair Average in Table 7 you want to verify.

 

In Facets,

"Output Files Menu"

"Residuals File"

"Select fields to output"

"Uncheck all"

Check - observation, expected value, element numbers

OK

Output to Excel.

 

In Excel, sort on the element numbers for your Facet.

Delete all rows except those for your element of your Facet.

 

Count and Sum the observations for your element of your Facet.

The count and sum should agree with Facets Table 7.

Average the observations. This should agree with the Table 7 Observed Average.

 

Now for the Fair Average:  This is the expected value for your element when it encounters elements of mean difficulty (usually 0) in all the other facets. Extreme-score elements are excluded from computing the mean.

 

From Facets Table 7, choose elements near the mean of the other facets.

 

In the Excel table, discover the expected values for your element combined with the mean elements of the other facets. The expected values should agree with the Table 7 Fair Average.

 


 

Problems with the Fair Average

 

If the "Fair averages" do not monotonically increase with the element measures, then

the misalignment of fair score with Rasch measures can occur when some items/task/raters etc. have different rating scales to other items/task/raters and all candidates are not rated on all rating scales.

 

For instance,

you do task 1 which has a rating scale from 0-10

but I do task 2 which has a rating scale from 0-5

 

We both have a measure of 2.00 logits.

 

Then your "fair score" will be 8.2 on the 0-10 item

But my "fair score" will be 4.1 on the 0-5 item

 

To get around this,

1. Write out an Anchorfile= from the Facets analysis.

2. Construct dummy data in which every candidate has a rating on every task (it doesn't matter what the value of the rating is).

3. Analyze the dummy data

4. The "fair score" for each candidate will be averaged across all the tasks.


Help for Facets Rasch Measurement and Rasch Analysis Software: www.winsteps.com Author: John Michael Linacre.
 

Now in progress: The Batchelorette - Australia - 2018: Rasch Measurement of Romance

Facets Rasch measurement software. Buy for $149. & site licenses. Freeware student/evaluation download
Winsteps Rasch measurement software. Buy for $149. & site licenses. Freeware student/evaluation download

State-of-the-art : single-user and site licenses : free student/evaluation versions : download immediately : instructional PDFs : user forum : assistance by email : bugs fixed fast : free update eligibility : backwards compatible : money back if not satisfied
 
Rasch, Winsteps, Facets online Tutorials

Forum Rasch Measurement Forum to discuss any Rasch-related topic

To receive News Emails about Winsteps and Facets by subscribing to the Winsteps.com email list,
enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Winsteps.com
The Winsteps.com email list is only used to email information about Winsteps, Facets and associated Rasch Measurement activities. Your email address is not shared with third-parties. Every email sent from the list includes the option to unsubscribe.

Questions, Suggestions? Want to update Winsteps or Facets? Please email Mike Linacre, author of Winsteps mike@winsteps.com

Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement with Raters and Rating Scales: Rasch Models for Rater-Mediated Assessments, George Engelhard, Jr. & Stefanie Wind Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez
Winsteps Tutorials Facets Tutorials Rasch Discussion Groups

 

Coming Winsteps & Facets Events
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Oct. 24 - 26, 2018,Wed.-Fri. Rasch workshop at Midwest Educational Research Association Annual Meeting, Cincinatti, Ohio (W. Boone, Winsteps), boonewjd@gmail.com
Jan. 25 - Feb. 22, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
April 4 - 8, 2019, Thur.-Mon. NCME annual meeting, Toronto, Canada https://ncme.connectedcommunity.org/meetings/annual
April 5 - 9, 2019, Fri.-Tue. AERA annual meeting, Toronto, Canada www.aera.net/Events-Meetings/Annual-Meeting
May 24 - June 21, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 28 - July 26, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 9 - Sept. 6, 2019, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 11 - Nov. 8, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 24 - Feb. 21, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 22 - June 19, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 26 - July 24, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 7 - Sept. 4, 2020, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith,Facets), www.statistics.com
Oct. 9 - Nov. 6, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 25 - July 23, 2021, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com

 

Our current URL is www.winsteps.com

Winsteps® is a registered trademark