Hi,
I am struggling to set up a data file for Facets – partly due to the fact that I am not sure my data are actually amenable for this analysis.
Very briefly about the data: Each candidate is marked twice, by one ordinary marker, and then by a senior marker. There is no overlap between candidates marked by ordinary markers, i.e. every ordinary marker has a different allocation of candidates. The (potential) link is provided by the senior markers. Each senior marker remarks the scripts marked by several ordinary markers. I don’t have item level marks but just grades (pass or fail) at scrip level.
The actual scripts are different between candidates (I think there are several versions between them), but I have ignored this for the trial analysis as I am only interested in seeing differences in marker severity and reliability.
I am not sure that such a set up will provide enough linking for a Facets analysis… But I wanted to try and see if the connection works.
I have set up a spec file for 2 facets (markers and candidates), but the analysis cannot be executed. I have obviously specified something (probably more than one thing) wrongly (as you can see from the attached screen shot) and my attempt at data file spec.
I would be grateful for any advice regarding setting up this data file, and, indeed, whether such data can be analysed in this way in the first place.
Many thanks,
Milja

I was unable to attach the spec file in the previous post, so here it is.

Good so far, Milja.

Every element needs a number, see attached file.

Are your ordinary markers nested within senior markers? If so, you may need to anchor the senior markers.

Your data a very thin, so you may need to make a Bayesian adjustment to the analysis to make every element (=parameter) estimable.

Thanks for this, Mike.
I have amended the spec file (and also addedd another module marked by the same markers to bulk up the data a bit), and managed to get some results. I attach the output file. As you can see from the attached output file, only about half the markers' measures are estimable. Also, the correlations are very low and lots are negative, so not sure if it is safe to rely on the derived measures at all...

Also, not sure why there is the list of unspecified elements in the output file (elements 660-826)?? There are only 659 elements (candidates) in the spec file - not sure why Facets was expecting more than that...

Regarding the nesting of the markers - about half are nested within senior markers, but there are some where two different senior markers have remarked the same ordinary marker's allocation (I guess these are the cases where Facets was able to derive a measurement). I attach a pivot table with all the markers for you to see. These are operational data from a not very clever remarking design...

Could you point me to some literature to find out how to spcify Bayesian adjustment for the anlysis?

BTW, for some reason the attachment facility only allows me to attach one file per post... Is this a bug?

Many thanks for your help!

This is just so I can attach another file as per the previous post...

Some more thoughts, Milja.

1. If you want to attach multiple files, please zip them together into one file.

2. Please upgrade to the current version of Facets: 3.70.1.
Then rerun your analysis with the current version of Facets.

3. Bayesian adjustment. When the data are very thin (many inestimable), then we need to add reasonable "dummy" data. One approach is to add two more observations to every element of the target facet: a good performance on a very easy item and a bad performance on a very hard item. Anchor the very easy item at a very low measure. Anchor the very hard item at a very high measure. Then every measure will be estimable between the very hard measure and the very easy measure.

Thanks for your help, Mike. Unfortunately, I will have to wait for our IT dept to process the Facets update request.

Hi again,
I was wondering - if the data I have are raw total scores out of 25 on a paper for each candidate (rather than item level data), is it possible to use such data for a Facets analysis? Not sure which model to specify because the data are not a rating scale, and obviously not dichotomous. Does it make sense to conceptualise this as a rating scale and recode somehow?

Thanks again!

Milja, if there is only one score for each candidate, then Facets cannot be used effectively.

If there is more than one score for each candidate, then, statistically, those scores are rating-scale items.
The Facets model would look like:
?,?,?,R25

Thanks, Mike. Each candidate has two scores - provided by two markers - ordinary and senior. The candidates are nested within ordinary markers, however, there is some overlap provided via senior markers as the same senior markers will have double marked more than one ordinary marker's work.

May I just ask something else as I am struggling to translate this into Facets logic. So, my data are total marks on papers (rather than item marks). Supposing that all the candidates did the same paper, do I need the 'paper' facet specified at all? If I do, then there would be only 1 element, right? Which does not make much sense to me... So my facets would be markers, candidates and paper (?,?,?,R25).

Or do I just need two facets in this case, candidates and markers (?,?,R25)? This is what I was going to use because, given that I only have paper-level marks, no stats can be derived about items in terms of difficulty anyway.

Or am I not interpreting this properly?

Thanks again!

You are correct, Milja. Two facets are enough.

Thanks, Mike.
I have just installed the latest version of Facets and reanalysed a file that was coming up with an error on a previous version. You suggested running it on the new version should get rid of that. However, this has happened again. The problem is that the output contains a long list of unspecified elements which (e.g. for facet 1 67-107), but I only have 66 markers in my data and these are specified in the spec file. Not sure why Facets expects more than that. This is also the case with facet 2.
I attach the spec and the output file.
Do you have any suggestions how to sort this out?
Thanks.

Apologies, managed to identify what the problem was - it was in my data, as might have been expected. Sorry again for wasting your time on this.

Hi,
I have been playing around with some data trying a bias analysis. I got an error message from Facets:
Facets 3.70.1 failed at Facets 23150:9
Subscript out of range

The message also asked for this to be reported to Winsteps.com, which I will do, but I also wanted to post here in case this is due to a mistake in my spec file, which is attached.

Grateful for any suggestions,
Milja

Apologies for the failure in Facets, Milja.

The problem is mismatch between the element identifiers and the data. Please try this:

Delements = NL
1,Marker
200037= UK200037
200073=     UK200073
200319=     UK200319
201116=     UK201116
205576=     UK205576
205594=     UK205594
205595=     UK205595
205599=     UK205599
205603=     UK205603
205604=     UK205604
205607=     UK205607
205609=     UK205609
205610=     UK205610
205613=     UK205613
205614=     UK205614
205617=     UK205617
205626=     UK205626
205645=     UK205645
205655=     UK205655
205661=     UK205661
205668=     UK205668
205670=     UK205670
205672=     UK205672
205677=     UK205677
205696=     UK205696
205716=     UK205716
205743=     UK205743
205791=     UK205791
205801=     UK205801
205842=     UK205842
205850=     UK205850
205914=     UK205914
206096=     UK206096
206100=     UK206100
206164=     UK206164
206167=     UK206167
206176=     UK206176
206179=     UK206179
206182=     UK206182
206187=     UK206187
206188=     UK206188
206194=     UK206194
206209=     UK206209
206316=     UK206316
206323=     UK206323
206324=     UK206324
206326=     UK206326
206331=     UK206331
206339=     UK206339
206348=     UK206348
206350=     UK206350
206352=     UK206352
206388=     UK206388
206389=     UK206389
206515=     IN206515
206516=     IN206516
206517=     IN206517
206519=     IN206519
206520=     IN206520
206523=     IN206523
206524=     IN206524
206710=     IN206710
206719=     IN206719
206720=     IN206720
206721=     IN206721
206726=     IN206726
206728=     IN206728
*

also

3,Country, A
1,UK,0
2,IN,0
*

Dvalues =     
3, 1, 1, 2     ; Marker country elements for facet 3 are indicated in the label of Facet 1, column 1 length 2
*