Decision Consistency

Assess.com discusses What is Decision Consistency? They write: "Decision consistency [DC] is an estimate of how consistent the pass/fail decision is on your test.  That is, if someone took your test today, had their brain wiped of that memory, and took the test again next week, what is the probability that they would obtain the same classification both times?"

 

Facets does not report a DC statistic directly, but Wan et al. (2007) describe how DC statistics are verified by means of data simulation. Accordingly an MFRM DC statististic can be computed directly by simulation. The procedure is

 

1.) Rasch-analyze all the data. Identify candidates above and below the cut-score.

 

2.) Use the Rasch results to simulate and analyze 100 hypothetical datasets matching the observed dataset.

 

3.) From analyses of the simulated data, compute the average % of candidates for whom the pass-fail decision is the same as for the observed data. This is the proposed Decision Consistency statistic. We expect Decision Consistency to be near 100% - that's the idea!

 

Details of the Decision Consistency procedure:

1.Use a dataset you already have to try this out.

2.Start with 2 simulated data files to verify this procedure works for you.

3.Do it with a loop in a batch file to perform 2 data simulations and analyses. Output 2 Scorefiles.

4.Use NotePad++ or similar to rectangular copy the candidate scores from the original analysis score file into Excel column 1. This is the master list. Followed by columns of scores for each of the simulations.

5.Whatever the cutscore, use an Excel =if formula to put 1 for pass, 0 for fail in column 1 of Worksheet 2 for the candidates.

6.Do the same thing for the simulated scorefiles in columns 2 and 3. So you have 3 columns of candidate scores in Worksheet 1. 3 columns of 1/0 in worksheet 2.

7.In Worksheet 3, column 2 put 1 if worksheet 2  column 1 =if with worksheet 2 column 2. otherwise 0 for each row

8.In Worksheet 3, column 3 put 1 if worksheet 2  column 1 =if with worksheet 2 column 3 otherwise 0  for each row

9.Decision consistency % is the average of the numbers in Worksheet 3 * 100%

10.When this procedure works for you, repeat with 100 simulated data files

 

Note: This can be streamlined using a VBA macro so that we don't need Worksheets 2 and 3, but to start off, it is easier to check that all is correct if we can see the intermediate results.

 

CASMA Research Report Number 22, Estimating Classification Consistency for Complex Assessments, Lei Wan, Robert L. Brennan, Won-Chan Lee, January 2007