Decision consistency

Method 1: For "decision consistency" we are not interested in overall reliability, we are interested in a cut-point. We need to know what is the probability that each person will be above or below the cut-point.

 

Use 100 simulations in Batch= mode. For each person, we compute its frequency of being above the cut-point (score or measure) in all simulations, and its frequency of being below the cut-point. The higher of these two frequencies is the expected decision for each person. We can then use these frequencies to compute the probability of the expected decisions across all simulations:

 

A. Probability of expected decision for one person = maximum ( frequency above cut-point, frequency below cut-point ) for the element / (count of simulations)

 

B. Decision consistency = Probability for all elements = sum(A) / count of persons

 

If the cut-point is in the tail of theperson distribution, then the Decision Consistency will be high. If the Cut-Point is in the center of the distribution it will be lower.

 

Example: you have an item bank in which the items have their Rasch difficulties based on earlier analyses..

 

For each new test form, anchor their items at the item bank difficulties and analyze their new data. This will tell you two things:

1  Items that have displacements. These items have drifted from their anchor values. Output IFILE= to Excel

2. Apply Method 1.

 

Second analysis: unanchor all items with noticeable displacements. Analyze the data again

1. Output IFILE= to Excel.

2. Apply method 1.

 

From 1. outputs: plot item difficulties by item number. This will show you how much some items have drifted.

 

From 2. how much the overall decisions have changed based on the cut-point.

 

Using this information, you can talk about decision consistency, whether it is based on raw score or logit measure.