1. If all facets are positive, higher raw score = more person ability. Higher raw score = more rater leniency. Higher raw score = more task easiness.
Positive bias/interaction means that the observed score is higher than expected.
Suppose we are modeling task-by-rater interaction, and the interaction is positive (score higher than expected)
Then if we think that the interaction is caused by the task, then the "absolute measure" of the task = task easiness + interaction. "Absolute measure" of the rater = rater leniency.
Then if we think that the interaction is caused by the rater, then the "absolute measure" of the task = task easiness. "Absolute measure" of the rater = rater leniency + interaction.
2. If some facets are positive and some facets are negative, then the relationship between raw scores, measures and interactions becomes complicated. Plots and reports differ depending on whether we model facets to be positive or negative, interactions to be positive or negative and to which facet we want to add the interaction. Example:
Positive: higher raw score = more person ability.
Positive; Higher raw score = more rater leniency.
Negative: Higher raw score = less task difficulty.
We must choose: Positive bias/interaction means that the score is higher than expected or lower than expected.
If we think of interaction for the rater, then we want more score = positive (more lenient) which add to rater leniency. If we think of interaction for the task, then we want more score = negative (more difficult) which adds to task difficulty making the task less difficult.
This becomes confusing. So I strongly recommend that all facets are positive if interactions can apply to both positive and negative facets.
After estimating the measures, Facets checks to see if any Model= specifications include Bias specifiers, "B". If so, for each such model, the specified Bias interaction is estimated for all the data (not just the data matching that particular model). Bias can be due to any type of interaction including Differential Item Functioning DIF, Differential Person Functioning DPF, Differential Rater Functioning DRF.
This is done by iterating through the data again and, after convergence, doing one further iteration to calculate statistics.
Computation of interactions is a two-stage process.
1. The measures for the elements, and the structure of the rating scale, are estimated. Then those values are anchored (fixed, held constant).
2. The expected values of the observations are subtracted from the observed values of the observations, producing the residuals.
3. The residuals corresponding to each interaction term (e.g., examinees rated by judge 4) are summed. If this sum is not zero, then there is an interaction.
4. The size of the interaction is estimated. A first approximation is:
Interaction (logits) = (sum of residuals) / (sum of the statistical-information in the observations).
Algebraically, first the Bn, Di, Cj, Fk are estimated using a Rasch model such as:
log ( Pnijk / Pnij(k-1)) = Bn - Di - Cj - Fk
Then the Bn, Di, Cj, Fk are anchored, and the bias/interaction terms, e.g., Cij, are estimated:
log ( Pnijk / Pnij(k-1)) = ( Bn - Di - Cj - Fk ) - Cij
Thus the Cij are estimated from the residuals left over from the main analysis. The conversion from residual score to bias interaction size is non-linear. Bias sizes may not sum to zero.
Bias, (also called interaction, differential item function, differential person function, etc.,) estimation serves several purposes:
1) in diagnosing misfit:
The response residuals are partitioned by element, e.g., by judge-item pairs, and converted into a logit measure. Estimates of unexpected size and statistical significance flag systematic misfit, focusing the misfit investigation.
2) in investigating validity:
A systematic, but small, bias in an item or a judge, for or against any group of persons, may be overwhelmed by the general stochastic component in the responses. Consequently it may not be detected by the usual summary fit statistics. Specifying a bias analysis between elements of facets of particular importance provides a powerful means of investigating and verifying the fairness and functioning of a test.
3) in assessing the effect of bias:
Since bias terms have a measure and a standard error (precision), their size and significance (t-statistic) are reported. This permits the effect of bias to be expressed in the same frame of reference as the element measures. Thus each element measure can be adjusted for any bias which has affected its estimation, e.g., by adding the estimate of bias, which has adversely affected an element, to that element's logit measure. Then the practical implications of removing bias can be determined. Does adjustment for bias alter the pass-fail decision? Does adjustment for bias affect the relative performance of two groups in a meaningful way?
4) in partitioning unexplained "error" variance:
The bias logit sample standard deviation corrected for its measurement error, can be an estimate of the amount of systematic error in the error variance (RMSE).
e.g., for a bias analysis of judges,
Bias logit S.D. = 0.47, mean bias S.E. = 0.32 (Table 13),
so "true" bias S.D. = (0.47² - 0.32²) = 0.35 logits,
but, this exceeds the RMSE for judges = 0.12 (Table 7).
Here, locally extreme judge-person scores cause an overestimation of systematic bias.
Adjusting for bias:
A straight-forward approach is to define the biased element as two elements: one element for one subset of judges (examinees, etc.) and a different items for the other subset of judges (examinees, etc.). This can done by defining an extra item element, and then adjusting item references in the data file accordingly.
Example:
Facets = 4 ; Items, candidates, examiners, bias adjustment
Non-center = 2 ; candidates float
Models =
?, 28, 17, 1, myscale ; allow for bias adjustment between candidate 28 and examiner 17
?, ?, ?, 2, myscale
*
Rating scale = myscale, R9
Labels=
1, Items
...
2, Candidates
...
3, Examiners
....
4, Bias adjustment, A
1, 28-17 adjustment ; the bias will be absorbed by this element, relative to element 2.
2, Everyone else, 0
*
Data=
1-5, 28, 17, 1, 1,2,3,4,5
1-5, 29, 23, 2, 5,4,3,2,1
.....
Non-Uniform DIF
Create a dummy facet for ability levels: then a three-way interaction between
person group dummy facet, ability level dummy facet, and item (or whatever).
Models=?,?B,?B,?,?B,R3 (or whatever)
Paired-Comparison Bias/Interaction Analysis:
Facets produces meaningful numbers in the Bias/interaction analysis when:
1) Use mirrored data, but set the weight = 1.0, instead of 0.5. For the main analysis, use weight 0.5.
2) Arrange the data so that the Models= is ..., -?,?,... instead of ...,?,-?,...
