﻿ Fair average

# Fair average

Meaning of the Fair Average:

The "Fair Average" is what the raters would have had if they had all rated the same "average" elements under the same "average" conditions. For instance, suppose one rater rated all the easy tasks (= high observed average rating)  but another rater rated all the hard tasks (= low observed average rating). Then the "Fair Average" says "What if both those raters had rated the same average task, what would the rating have been?". We can then use the Fair Average to compare the severity/leniency of the raters as though they had rated under the same conditions.

Example: What's the relationship between the raw score and Fair Averages?

They are both in the original rating-scale metric. The raw score on an item is the original observation. The Fair Average is the original observation adjusted for its context. Suppose that my performance receives a rating of 3 from a lenient rater. My Fair Average is 2.5. Your performance receives a rating of 3 from a severe rater. Your Fair Average is 3.5. Comparing your "3" with my "3" is unfair, because you had a severe rater (who generally gives low ratings) and I had a lenient rater (who generally gives high ratings). After adjusting for rater severity/leniency, our "Fair Average" ratings are 2.5 and 3.5. These give a fair comparison.

The "fair average" transforms the Rasch measure back into an expected average raw response value. This value is in a standardized environment in which all other elements interacting with this element have a zero measure or the mean measure of all elements in their facet. This is "fair" to all elements in the facet, e.g., this adjusts raw ratings for severe and lenient raters. This enables a "fair" comparison to be made in the raw score metric, in the same way that the measure does on the linear latent variable. Fair-M uses the facet element means as the baseline. Fair-Z uses the facet local origins (zero points) as the baseline. These are set by Fair average=.

The original purpose of Facets (in 1986) was to construct software that would automatically adjust for differences in rater severity/leniency. So this has been done for your data. No adjustment or trimming of usefully-fitting raters is necessary, regardless of their severity/leniency.

But Facets does assume that the average leniency of the raters is at the required standard of severity/leniency (usually by centering the rater facet at zero logits). If you see, from evidence external to the data, that the average leniency of the of the raters is too high or too low, then please

2. this facet has one element.

3. Anchor this element at the adjustment value

4. Include the element in all the observations in the dataset by using dvalues=

5. perform an analysis of all the data including the adjustment element The rater leniencies are now correct, but the fair averages are not correct for the adjusted leniencies.

6. output an anchorfile

7. replace the ? in the model specification with an X

8. analyze the anchorfile with the dataset

9. there will be displacements of the size of the adjustment value. The fair averages are correct for the adjusted leniencies.

This procedure will require several attempts before it produces the correct results.

Standard Error of the Fair Average:

Probably the easiest way to obtain statistically rigorous estimates of the S.E.s of the Fair Averages is to use Facets to simulate 20 or more datasets based on the current dataset. Then the S.D.s of the 20 or more replications of each Fair Average is the S.E. of the Fair Average.

We can approximate the S.E. of a Fair Average by using two elements from the Facet as reported in Table 7.

facet with similar characteristics (same number of ratings, similar rating situations) but a different measure.

S.E. of Fair score for element 1 =  S.E. of Measure for element 1 * (Fair score for element 1 - Fair score for element 2) / (Measure for element 1 - Measure for element 2)

Standard errors are always written as positive numbers.

Example: Anne and Chris were rated the same number of times in the same type of rating situations:

Anne: Fair average = 4.64  measure = -.07  S.E. of measure = .17

Chris: Fair average = 4.23  measure = -.25

S.E. of Anne's Fair average = .17 * (4.64 - 4.23) / (-.07 - -0.25) = 0.39 score points.

Calculation of the Fair Average Score

The observed average score is the average rating received by the element. The logit measure is the linear measure implied by the observations. This is adjusted for the measures of whatever other elements of other facets participated in producing the observed data. It is often useful to transform these measures back into the original raw score metric to communicate their substantive meaning. Fair Average does this. It is the observed average adjusted for the measures of the other elements encountered. It is the observed average that would have been received if all the measures of the other elements had been located at the average measure of the elements in each of their facets.

A basic many-facet Rasch model for observation Xnmij is:

log ( Pnmijk / Pnmij(k-1)) = Bn - Am - Di - Cj - Fk

where

Bn is the ability of person n, e.g., examinee: Mary,

Am is the challenge of task m, e.g., an essay: "My day at the zoo".

Di is the difficulty of item i, e.g., punctuation,

Cj is the severity of judge j, e.g., the grader: Dr. Smith,

Fk is the barrier to being observed in category k relative to category k-1,

where k=0 to t, and F0=0.

To compute the fair average for person n (or task m, item i, judge j), set all element parameters except Bn (or Am, Di, Cj) to their mean (or zero) values. Thus, the model underlying a fair rating, when Fair=Mean, is:

log ( Pnmijk / Pnmij(k-1)) = Bn - Amean - Dmean - Cmean - Fk

or, when Fair=Zero, it becomes:

log ( Pnmijk / Pnmij(k-1)) = Bn - Fk

and the Fair average is sum(k Pnmijk) across categories k=0 to t.

Example: Students are rated on 6 items. I want a fair-average for each student on each item.

1. Perform the analysis of all the data.

2. Output an Anchorfile=

3. Unanchor the students (remove the ,A for students in Labels=)

4. Analyze the anchorfile one item at a time by commenting out all items except 1 in Labels=

5. The reported measures and fair-averages for the students will be the fair averages for each item.

6. To assemble these fair-averages, output a Scorefile= from each one-item analysis to, say, Excel, selecting student identification and fair average fields.

Verifying the Fair Average

Analyze your data in Facets. Choose a Facet and an element whose Fair Average in Table 7 you want to verify.

In Facets,

"Uncheck all"

Check - observation, expected value, element numbers

OK

Output to Excel.

In Excel, sort on the element numbers for your Facet.

The count and sum should agree with Facets Table 7.

Average the observations. This should agree with the Table 7 Observed Average.

Now for the Fair Average:  This is the expected value for your element when it encounters elements of mean difficulty (usually 0) in all the other facets. Extreme-score elements are excluded from computing the mean.

From Facets Table 7, choose elements near the mean of the other facets.

In the Excel table, discover the expected values for your element combined with the mean elements of the other facets. The expected values should agree with the Table 7 Fair Average.

Problems with the Fair Average

If the "Fair averages" do not monotonically increase with the element measures, then

the misalignment of fair score with Rasch measures can occur when some items/task/raters etc. have different rating scales to other items/task/raters and all candidates are not rated on all rating scales.

For instance,

you do task 1 which has a rating scale from 0-10

but I do task 2 which has a rating scale from 0-5

We both have a measure of 2.00 logits.

Then your "fair score" will be 8.2 on the 0-10 item

But my "fair score" will be 4.1 on the 0-5 item

To get around this,

1. Write out an Anchorfile= from the Facets analysis.

2. Construct dummy data in which every candidate has a rating on every task (it doesn't matter what the value of the rating is).

3. Analyze the dummy data

4. The "fair score" for each candidate will be averaged across all the tasks.

Help for Facets Rasch Measurement Software: www.winsteps.com Author: John Michael Linacre.

 Forum Rasch Measurement Forum to discuss any Rasch-related topic

Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement with Raters and Rating Scales: Rasch Models for Rater-Mediated Assessments, George Engelhard, Jr. & Stefanie Wind Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez
Winsteps Tutorials Facets Tutorials Rasch Discussion Groups

Coming Rasch-related Events
Jan. 5 - Feb. 2, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 10-16, 2018, Wed.-Tues. In-person workshop: Advanced Course in Rasch Measurement Theory and the application of RUMM2030, Perth, Australia (D. Andrich), Announcement
Jan. 17-19, 2018, Wed.-Fri. Rasch Conference: Seventh International Conference on Probabilistic Models for Measurement, Matilda Bay Club, Perth, Australia, Website
Jan. 22-24, 2018, Mon-Wed. In-person workshop: Rasch Measurement for Everybody en español (A. Tristan, Winsteps), San Luis Potosi, Mexico. www.ieia.com.mx
April 10-12, 2018, Tues.-Thurs. Rasch Conference: IOMW, New York, NY, www.iomw.org
April 13-17, 2018, Fri.-Tues. AERA, New York, NY, www.aera.net
May 22 - 24, 2018, Tues.-Thur. EALTA 2018 pre-conference workshop (Introduction to Rasch measurement using WINSTEPS and FACETS, Thomas Eckes & Frank Weiss-Motz), https://ealta2018.testdaf.de
May 25 - June 22, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 27 - 29, 2018, Wed.-Fri. Measurement at the Crossroads: History, philosophy and sociology of measurement, Paris, France., https://measurement2018.sciencesconf.org
June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 25 - July 27, 2018, Wed.-Fri. Pacific-Rim Objective Measurement Symposium (PROMS), (Preconference workshops July 23-24, 2018) Fudan University, Shanghai, China "Applying Rasch Measurement in Language Assessment and across the Human Sciences" www.promsociety.org
Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Sept. 3 - 6, 2018, Mon.-Thurs. IMEKO World Congress, Belfast, Northern Ireland www.imeko2018.org
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com

Our current URL is www.winsteps.com