Investigating differences between rater groups

The Problem:

Videotapes of 2 different flight crews, consisting of 2 crew members each, the pilot-in-command (PIC) and the second-in-command (SIC), are rated on 10 items that measure technical and teamwork skills. 2 overall items assess crew technical and teamwork performance. There is one item for overall evaluation of the PIC, and one item for overall evaluation of the SIC. There are 3 groups of judges, each group consisting of 7 to 10 judges. Each judge rates both crews on all items (4-point Likert scale). I want to investigate the differences between the groups of judges.

 

Solution:

Here is the skeleton of a suggested Facets control file:

 

Title="Air Crew Videotapes"

 

; facet 1 is judges

; facet 2 is judge group (a dummy facet)

; facet 3 is crew number

; facet 4 is crew component: crew, PIC or SIC

; facet 5 is items

Investigating differences between rater groups

facets=5

 

models=

?,#,?,?,?,Likert     ; model each group to have its own rating scale (or partial credit)

?,?B,?,?,?B,R        ; look for group by item interactions

?,?B,?B,?,?,R        ; look for group by crew number interactions

*

 

rating scale = Likert,R4,Specific   ;we model each group to have its own scale

1=SD                                ; compare the rating scale (or partial credit) structures

2=D                                 ; reported for the 3 groups 

3=A

4=SA

*

 

labels=

1,Judges

1=Fred,,1          ;Fred is in group 1: Note the ,, to skip over the pre-set logit value field.

2=Mary,,3          ;Mary in group 3

3=Alfonso,,2       ;Alfonson in group 2

....               ;there will be subtotals by group

*

2,Group,A   ;dummy facet - used for classification

1=Group A,0 ;anchored at 0 to remove for measurement,

2=Group B,0 ;group facet used for classification and summary fit statistics

3=Group C,0 

*

 

3,Crew number ; the videotapes

1=Crew X

2=Crew Y

*

4,Component

1=Crew      ;used for items 1-12

2=PIC       ;used for item 13

3=SIC       ;used for item 14

*

5,Items

1=technical

....

10=teamwork

11=technical performance

12=teamwork performance

13=PIC overall 

14=SIC overall 

*

data=      ;here are some example data lines:

2,3,1,1,1-12, 2,1,3,2,4,1,2,1,3,2,1,2 ;Mary in group 3 rates crew 1's crew on items 1-12

2,3,1,2,13,   4                   ;Mary in group 3 rates crew 1's PIC on item 13

2,3,1,2,14,   3                   ;Mary in group 3 rates crew 1's SIC on item 14

 

Here is the skeleton of a likely Facform control file:

Facform can be downloaded from www.winsteps.com/facets.htm

 

Original data format (say) in file rawdata.txt:

judge id     cols 1-3

judge group  cols 4

Crew number  col  5

Item ratings cols 6-19   (14 items)

 

$Title = "Format the Air Crew judge data"

$Input = "rawdata.txt"

$Output= "facdata.txt"

 

$Facets=5                   ;define number of facets

$Flabel=1,"Judges"          ;define facet names

$Flabel=2,"Group,A"

$Flabel=3,"Crew number"

$Flabel=4,"Component"

$Flabel=5,"Items"

 

$Label=1,$S1W3             ;identification for facet 1 judge

$Label=2,$S4W1             ;identification for facet 2 group

$Label=3,$S5W1             ;identification for facet 3 crew number

 

$label=4,1,"Crew"          ; identification for facet 4 - the crew

item = 1

$Do=12

    $Label=5,item          ; identification for facet 5 - item number

    $Rating=$S(item+5)W1   ; response to item 1 etc.

    item = item +1         ; next item

$Again

 

$label=4,2,"PIC"          ; identification for facet 4 - PIC

$Label=5,13,"PIC overall" ; identification for facet 5 - item 13

$Rating=$S18W1            ; response to item 13

 

$label=4,3,"SIC"          ; identification for facet 4 - SIC

$Label=5,14,"SIC overall" ; identification for facet 5 - item 14

$Rating=$S19W1            ; response to item 14