Mike,

We recently received the results from a survey attempting to measure parent satisfaction with their child's school. I did not design the survey but have been asked to report on the findings. About 80% of our students are classified as economically disadvantaged and about 35% have a primary language other than English spoken at home. Our communications department wanted to provide parents and guardians with the option to take the survey online or use a paper scan sheet because many of our families simply can't afford a computer.

I've attached the results with the items in misfit order as well as some very puzzling results with regard to dimensionality. At the end of the attachment are the results from the t-test. With extreme scores included, there is clearly a difference in responses between those families using online versus paper. But once the extreme scores are removed, there does not seem to be a significant difference.

Wanting to investigate further, I calibrated the items using all students (566), then only those that took it online (193) then those that handed in paper scan sheets (373). The dimensionality tests for each run are very different, though all three seem to indicate a 2nd dimension of some kind. But each reports this as something different.

For example, when all parents were used or only those using paper scans, the 2nd dimension seems to have something to do with "This School" items. However, when only the online parent data were used, the 2nd dimension seems to be centered on staff.

I would very interested in your thoughts about this.

Uve, this is an interesting sociological investigation

Let me speculate ....

Thinking of parents I know, the online version would appeal to younger, more technological, more affluent, more "English" families. The paper-and-pencil version would appeal to parents with opposite characteristics.

From your findings, this suggests that "paper" parents see the "School" as a special topic, but the online parents see the "Staff". Perhaps the difference is that more traditional parents want their children to go to a good school (the teacher is secondary), but more modern parents want their children to be taught by a good teacher (the school is secondary).

My own parents were "modern" for their time, and they focused on good teachers ahead of good schools. Their choice for me has been justified. The school with good teachers is now ranked ahead of the good school they would have chosen!

Uve, undoubtedly you can do better than these speculations ....

Thanks Mike. For me personally, it's often hard to be objective about my own data which comes from the world in which I'm surrounded. But yes, I agree, with your observations.

For me there's an awkward crossroads of sorts. When we engage in dimensionality analysis and get different results depending on whether we are looking at all respondents or primary subgroups, then I want to make sure as best as is possible that I am not running down too many rabbit holes. In other words, in this example, I feel reality changes significantly depending on how the data is being disaggregated. Decisions about what to do or how to modify the survey could be contradictory based which hole I chase the rabbit down.  

Again, thanks for taking your valuable time and going over this.