Table 13 is the bias calibration report

This Table compares the local (biased?) measure of one element with its measure from the main analysis. Zscore=, Bias=, Xtreme=, Arrange= and Juxtapose= control this Table.

Ratings of Scientists (Psychometric Methods p.282 Guilford 1954) 06-10-2007 08:37:12

Table 13.1.1 Bias/Interaction Calibration Report (arranged by mN).

Bias/Interaction: 1. Senior scientists, 2. Junior Scientists (higher score = higher bias measure)

-----------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------

| 25 17.3 5 1.54| .71 .29 2.43 | .3 .3 | 11 2 Brahe .24 4 David -.46 |

| 14 27.0 5 -2.60| -1.27 .36 -3.55 | .7 .6 | 14 2 Brahe .24 5 Edward .42 |

-----------------------------------------------------------------------------------------------------------

| 24.2 24.2 5.0 .00| .00 .32 .02 | .6 .6 | Mean (Count: 21) |

| 6.9 4.9 .0 .96| .47 .02 1.41 | .4 .4 | S.D. (Population) |

| 7.1 5.0 .0 .98| .48 .02 1.44 | .4 .4 | S.D. (Sample) |

-----------------------------------------------------------------------------------------------------------

Fixed (all = 0) chi-square: 41.6 d.f.: 21 significance (probability): .00

-----------------------------------------------------------------------------------------------------------

The column headings mean:

Obsvd Score =

raw score of the estimable responses involving these elements simultaneously, as observed in the data file.

Exp. Score =

expected score based on the calibrations from the main analysis.

Obsvd Count =

number of estimable responses involving these elements simultaneously.

Obs-Exp Average =

observed score less the expected score divided by observed count, the bias in terms of the response metric.

Bias Size =

Size of bias measure in log odds units relative to overall measures. Only large or significant biases are listed greater than Zscore=. For clarification, compare the ranking of the Obs-Exp Average with that of the Bias Size. In this case, larger observed scores correspond to higher Bias sizes, i.e., higher abilities, higher leniencies, higher easiness.

For (measure+bias), add Bias Size to the element "measr", or subtract Bias Size from "measr". Addition for persons that are locally more able. Subtraction for items that are locally more difficult. Look at the "Obs-Exp Average".

Model Error =

standard error of the bias estimate.

t =	Student's t-statistic testing the hypothesis "There is no bias apart from measurement error". The "Obsvd Count"-2 approximates the degrees of freedom of the t-statistic. With many observations, the t-statistic approximates a normal distribution with mean = 0, S.D. = 1, i.e., a z-score.

Infit MnSq and Outfit MnSq =

Does the bias explain all the misfit or is there also another source of misfit? Values are expected to be less than 1.0 because the bias is explaining some of the overall misfit. These statistics do not report the fit of the bias terms.

The t-statistic is the report of a test of the statistical significance of the size of the bias. The mean-square fit statistics do not report on whether there is bias or not. They report on how much misfit there is in the data after the bias is removed. With the inclusion of bias terms, the model is overparameterized, so it is expected that the data will overfit the model. The purpose of the mean-square fit statistics is to help you determine whether the misfit in the data is explained by the bias or is due to other causes.

For each facet entering into the bias calculation:

Sq =	a sequence number used to reference the bias term - useful for referring to a specific line in this Table.

N =	element number with Facet

Senior Sc =

Name of facet: elements listed below

measr =

Measure of element from main analysis.

In the summary statistics,

Count =

the number of modelled bias terms found in the data.

S.D. (Population) =

the standard deviation if this sample is the whole population

S.D. (Sample) =

the standard deviation if this sample is a random sample from the whole population

Fixed (all=0) chi-square =

A test of the "fixed effect" hypothesis: "Can this set of interactions be regarded as sharing the same measure of approximately 0.0 after allowing for measurement error?" The chi-square value and degrees of freedom (d.f.) are shown. The significance is the probability that this "fixed" hypothesis is the case. This is not a test of "Can these interactions be disregarded?" Individual interactions may be large and significant. For instance, one bad tire in a production run of 1000 tires may not indicate a "statistically significant" problem in the production process, but I still don't want it on my car!