Dimensionality investigation - an example

Michael Commons asked: "What in Winsteps tells how unidimensional a fit is?"

 

The Rasch measures estimated by unidimensional Rasch models are forced to be unidimensional. Off-dimensional aspects of the data are in the part of the data not explained by the Rasch measures, i.e., the Rasch residuals. The Rasch residuals decompose into (a) the randomness predicted by the Rasch model, and (b) components on dimensions other than the unidimensional Rasch variable, (c) off-dimensional  noise, such as random guessing.

 

In empirical data, (a) and (c) usually dominate (b), so that item-level or person-level fit statistics tend to be insensitive to multidimensionality, as R.P. McDonald (1985) reports. Accordingly we must focus on techniques that quantify (b), such as PCA of residuals. If the eigenvalues reported by PCA approximate the size predicted by the Rasch model, then the data are effectively unidimensional. Otherwise, the bigger the eigenvalues, the less unidimensional are the data.

 

Roderick P. McDonald, Factor Analysis and related methods. Hillsdale, NJ: Lawrence. Erlbaum Associates, 1985.

 


 

Multidimensionality is complicated, because it depends on the purpose of the instrument.

 

Here is a multidimensional instrument. It has 5 items: 1 geography item. 1 history item. 1 cooking item. 1 carpentry item. 1 arithmetic item.

This instrument is multidimensional, but none of its dimensions is big enough to make a unidimensional instrument. We cannot split this multidimensional instrument into unidimensional instruments, but that does not make the instrument unidimensional. However, we may decide that this instrument is measuring a general "education" variable, and declare that this instrument is unidimensional for our purposes.

 

For instance, an arithmetic test (addition, subtraction, multiplication, division) is unidimensional from the perspective of school administrators deciding whether a child should advance to the next grade-level, but the same test is multidimensional from the perspective of the school psychologist diagnosing learning difficulties. For instance, learning difficulties with subtraction in young children may indicate social maladjustment.

 

Also

 

Here is an example. We can proceed as follows:

 

a. Compare the Raw Variance explained by items (19.8%) with the Unexplained variance in 1st contrast (7.1%). Is this ratio big enough to be a concern? In your analysis, the Rasch dimension dominates (almost 3 times the secondary dimension), but the secondary dimension is noticeable.

 

b. Is the secondary dimension bigger than chance? Eigenvalue = 2.8. This is the strength of 3 items. We do not expect a value of more than 2 items by chance. www.rasch.org/rmt/rmt191h.htm - and we would also need at least 2 items to think of the situation as a "dimension" and not merely an idiosyncratic item.

 

c. Does the secondary dimension have substance? Looking at your plot, we can see that items ABCDE are separated vertically (the important direction) from the other items. They are the core of the secondary dimension. 5 items are enough items that we could split them into a separate instrument (exactly as we could with "subtraction" on an arithmetic test).

Is this secondary dimension important enough, and different enough, that we would consider reporting two measures (one for ABCDE and one for the other items) rather than one measure for all items combined? The content of ABCDE appears to be psycho-social (e.g., one item includes the word "anxious" in this example). The other items are more physical (e.g., one item includes the word "walking" in this example).  Consider the purpose of the instrument. Is "anxious" important or not?  Is it part of the central purpose for the instrument? Would the instrument be improved or degraded (from a usefulness perspective) if items ABCDE were omitted? Would the instrument be improved or degraded (from a usefulness perspective) if a separate measure was reported for items like ABCDE?

 

d. Rasch-analyze the sample on the ABCDE items and then on the other items. Cross-plot the person measures.

Look at the correlation of the two sets of person measures (and the correlation disattenuated for measurement error). Is the correlation noticeably low? In this example, the disattenuated correlation was 0.82, indicating that the dimensions share explains about 67% of the person measure variance.

We expect most people to lie along a statistical diagonal. Who is off-diagonal? (Perhaps the people with social problems.) Are they important enough to merit a separate measurement system? For instance, on an English-language test, native-speakers of English, and second-language speakers usually have different profiles. Native speakers speak relatively better. Second-language speaker may spell relatively better. But two measures of English-language-proficiency are rarely reported.

 

If you decide that the secondary dimension is important enough to merit two measures, or the secondary dimension is off-dimension enough to merit omitting its items, then the instrument is multidimensional (in practice). If not, then the instrument is unidimensional (in practice), no matter what the statistics say.

 

Tentative guidelines based on the % of the sample are sampling dependent. If you are planning to apply a criterion such as "5% of the sample", then verify that your sample matches the intended target population of the instrument. In general, 5% seems very low. Would we institute a special measurement system for 1 child in a classroom of 20 children?  Unlikely? We would probably need at least 4 children = 20% before we would consider reporting (and acting on) two measures.

 

In the USA, African-Americans comprise 13% of the population, and there is a debate about whether or not they should have special measurement systems. In some situations they do. And, similarly, whether there should be special provision for Spanish-speakers (15% of the USA population). In some situations there are. These percentages suggest that a threshold of about "10% of the sample" may be reasonable for separate measurement procedures.

 

My conclusion about this instrument (knowing nothing about its practical purpose) would be that the instrument is multidimensional and that items ABCDE should be omitted (or rewritten or replaced to emphasize their physical rather than their psychological aspects).

 

"Unidimensionality" is a choice based on the circumstances, so, if you are writing a paper, then please include a discussion of why (or why not) you decided that the instrument is multidimensional. This would be helpful to other researchers.

 

     Table of STANDARDIZED RESIDUAL variance (in Eigenvalue units)

                                                 -- Observed --    Expected

Total raw variance in observations     =         39.8 100.0%         100.0%

  Raw variance explained by measures   =         18.8  47.2%          48.0%

    Raw variance explained by persons  =         10.9  27.4%          27.8%

    Raw Variance explained by items    =          7.9  19.8%          20.1%

  Raw unexplained variance (total)     =         21.0  52.8% 100.0%   52.0%

    Unexplned variance in 1st contrast =          2.8   7.1%  13.5%

    Unexplned variance in 2nd contrast =          2.6   6.5%  12.3%

    Unexplned variance in 3rd contrast =          2.1   5.4%  10.2%

    Unexplned variance in 4th contrast =          1.7   4.4%   8.3%

    Unexplned variance in 5th contrast =          1.6   4.0%   7.6%

 

1st contrast:

 


 

Question: My instrument has 200 items and my First Contrast has an eigenvalue of 9.2. Could this be explained by the large number of items analyzed?

 

Answer: Yes, the more items there are, then the more likely that a random cluster of items are inter-correlated by accident.

 

So, the first step with the potentially 9 or so items in a secondary dimension is to look at the plot in Winsteps Table 23.1. Is there a cluster of items at the top or bottom of the plot that share a content area that differs noticeably from the other items?

 

If so, this could be a secondary dimension. So, the next thing is to look at the table of correlations in Table 23.0. This reports the correlations of the person measures on the different clusters of items identified in the first contrast. Is the disattenuated correlation between the person measures on the suspect cluster of items and the person measures on the other items low or negative?

 

If so, the suspect cluster of items is measuring something different. It is a different dimension. When items on this secondary dimension are included with the items on the dominant dimension, then responses to the secondary items look like random noise from the viewpoint of the dominant dimension.

 

If not, the suspect cluster of items is probably a secondary strand in the content area, similar to "word problems" on an arithmetic test.  Performance profiles of the person across strands will not be even, but will also not be contradictory.

 

The "variance explained" can be somewhat misleading because it is dominated by the variance of the item difficulties and the variance of the person abilities. We may be contrasting a wide range of abilities and/or difficulties on the dominant dimension against a narrow range of abilities and/or difficulties on the secondary dimension.


Help for Winsteps Rasch Measurement Software: www.winsteps.com. Author: John Michael Linacre

For more information, contact info@winsteps.com or use the Contact Form
 

Facets Rasch measurement software. Buy for $149. & site licenses. Freeware student/evaluation download
Winsteps Rasch measurement software. Buy for $149. & site licenses. Freeware student/evaluation download

State-of-the-art : single-user and site licenses : free student/evaluation versions : download immediately : instructional PDFs : user forum : assistance by email : bugs fixed fast : free update eligibility : backwards compatible : money back if not satisfied
 
Rasch, Winsteps, Facets online Tutorials

 

Forum Rasch Measurement Forum to discuss any Rasch-related topic

Click here to add your email address to the Winsteps and Facets email list for notifications.

Click here to ask a question or make a suggestion about Winsteps and Facets software.

Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement with Raters and Rating Scales: Rasch Models for Rater-Mediated Assessments, George Engelhard, Jr. & Stefanie Wind Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez
Winsteps Tutorials Facets Tutorials Rasch Discussion Groups

 


 

 
Coming Rasch-related Events
Jan. 5 - Feb. 2, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 10-16, 2018, Wed.-Tues. In-person workshop: Advanced Course in Rasch Measurement Theory and the application of RUMM2030, Perth, Australia (D. Andrich), Announcement
Jan. 17-19, 2018, Wed.-Fri. Rasch Conference: Seventh International Conference on Probabilistic Models for Measurement, Matilda Bay Club, Perth, Australia, Website
Jan. 22-24, 2018, Mon-Wed. In-person workshop: Rasch Measurement for Everybody en español (A. Tristan, Winsteps), San Luis Potosi, Mexico. www.ieia.com.mx
April 10-12, 2018, Tues.-Thurs. Rasch Conference: IOMW, New York, NY, www.iomw.org
April 13-17, 2018, Fri.-Tues. AERA, New York, NY, www.aera.net
May 22 - 24, 2018, Tues.-Thur. EALTA 2018 pre-conference workshop (Introduction to Rasch measurement using WINSTEPS and FACETS, Thomas Eckes & Frank Weiss-Motz), https://ealta2018.testdaf.de
May 25 - June 22, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 27 - 29, 2018, Wed.-Fri. Measurement at the Crossroads: History, philosophy and sociology of measurement, Paris, France., https://measurement2018.sciencesconf.org
June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 25 - July 27, 2018, Wed.-Fri. Pacific-Rim Objective Measurement Symposium (PROMS), (Preconference workshops July 23-24, 2018) Fudan University, Shanghai, China "Applying Rasch Measurement in Language Assessment and across the Human Sciences" www.promsociety.org
Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Sept. 3 - 6, 2018, Mon.-Thurs. IMEKO World Congress, Belfast, Northern Ireland www.imeko2018.org
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com

 

 

Our current URL is www.winsteps.com

Winsteps® is a registered trademark
 


 
Concerned about aches, pains, youthfulness? Mike and Jenny suggest Liquid Biocell