Mike,

I've attached a modification I created from the Winsteps paired Excel analysis output of a test containing 45 items and over 800 students. While examining a pair of examinees who had 20 identical wrong items incorrect for choosing the same distractor options (Table 35.4) I discovered that the pair did not have the same teacher. How would it be possible for two different students to take the same test in two different classrooms and get almost half of the questions wrong by marking the same distractors?  

Perhaps they were communicating by means of some electronic device, like a phone. Teachers are extremely vigilant during testing to check for this and monitor the class, but I think it is very easy to underestimate the ability of some students to utilize them even under the most strict circumstances.  I know of one situation where the match occured because the site data entry person scored the same test under two different student ID numbers by mistake. So there can be many other factors that might explain something like this.

However, if we assume no data errors occured and no communication occured between these two examinees, then it seems we are only left with the conclusion that this was mere chance. This chance was extremely low--they likey had a better chance of winning the lottery. So I know that I can ignore this data because the examinees did not have the same teacher. It was then just a freak circumstance of chance--someone was struck by lightning or a meteorite.

Now, if you choose the Same Teacher = Yes in the table filter, you'll notice that there is one pair of examinees that has 22 matching incorrect items. I think the next question is obvious: how sure can I be that this isn't chance as well?  

I guess I'm wanting to be more sure about avoiding false postives, but I'm a bit shaken by the data that I see when I choose Same Teacher = No. Perhaps my probability calculations used to determine the odds of the matches occuring are incorrect in some manner. Could you confirm that my math is right? I would rather this be result of poor math than the nature of probability, which I have a feeling is how you're likley to answer.  

I had to use Excel 2010 because there are almost over 90,000 rows. If you are using 2003, it will open but you will not be able to manipulate the pivot table and it will clip off all rows after 65,000.

Thanks as always for your comments.

Uve, please try this: http://www.rasch.org/rmt/rmt61d.htm - when we first did that analysis, it was ground-breaking

Yes, I do like this resource and have read through it before.

DNA evidence in a murder trial usually supports conviction because the odds exceed Earth's population, i.e. 1 in 60 billion match. Evidence tampering, identical twin, lab errors, etc. ruled out, it is impossible for someone else to have the identical DNA.

Now suppose the Earth's population exceeds 100 trillion, the same odds could no longer help support conclusive conviction and the test would carry much less weight.

I guess that's where I'm going with my question. Given the large number of items and the very large number of respondents, a much greater than 1 in 1 million chance in my data seems to be a very freak, yet fairly conclusive false positive if we rule out communication between classrooms or a clerical error.

Like my example of the DNA test, perhaps the emprical data needs another measure of some kind. Something that better balances odds with total population.

Yes, we have to be careful about being "fooled by randomness" (Nassim Nicholas Taleb). The "Hitchhiker's Guide to the Galaxy" is based on the premise that, if we could discover the exact probability of an unlikely event, then we could engineer the situation where that probability becomes reality!

This was part of the original motivation for http://www.rasch.org/rmt/rmt61d.htm . Another statistical consultant presented a probability analysis, but it was found to be confusing, rather than convincing. Expert witnesses could not refute the argument: "this is an unlikely event, but it can happen, and this is the instance where it did happen." A non-probabilistic approach was required.

As you remark, DNA evidence may also find itself in this same situation before too much longer. My own analysis of DNA datasets makes me skeptical of the huge improbabilities of coincidences that are claimed. My suspicion is that the DNA sample spaces for the probability computations include many DNA combinations that cannot be observed.

Mike,

I was trying to find a response to an earlier thread I posted related to this one but couldn't. I just wanted to confirm the calcuations for determining probability given a situation in which two respondents have 12 items marked wrong using the same distractor (TWOLOW in the paired comparison Excel output table) on a MC test with four distractors total . Here's an example. Is the formula correct?

((0.25^2)+(0.25^2)+(0.25^2))^12  = 0.000000001888058

Also, if all is correct, I could interpret the output as less than 1 in 10 million of being chance. Would that also be correct?

Corrected: Uve, assuming the distractors are equally probable, then the chances that two persons would choose the same distractor for one item = 4*(0.25)^2, and so for 12 items = (4*(0.25^2))^12

Thanks!

Uve, please see my correction. I was thinking of selecting the same specific option, e.g., Option 1, but, of course, the agreement can be between any of the 4 options for any of the 12 items.

Probabilities are slippery to compute. It is easy to imagine the wrong sample space