Responses to previously calibrated items administered in a computerized adaptive testing (CAT) mode may be used to recalibrate the items. This live-data simulation study investigated the possibility, and limitations, of on-line adaptive recalibration of precalibrated items. Responses to items of a Rasch-based paper-and-pencil licensure examination were used to simulate CAT and paper-and-pencil administrations, defining CAT forms of varying difficulty levels and samples of various sizes. Forms were calibrated and new b-values were compared with the bank b-values obtained from responses of the reference group taking the paper-and-pencil examination. Results indicate that bank b-values were not well replicated when difficult items were calibrated using responses from able examinees and easy items were calibrated using responses from less able examinees. On-line adaptive recalibration as simulated in this study has limitations. However, a "modified" on-line adaptive recalibration may still be a possibility as long as: (1) a reasonably large CAT recalibration sample is predefined from the reference group; (2) the sample has a mean ability similar to that of the reference group; and (3) items to be recalibrated together are relatively heterogeneous in Rasch difficulty. Eight tables are included. (Contains 3 references.) (Author/SLD)
Authors
- Peer Reviewed
- F
- Publication Type
- ['Reports - Evaluative', 'Speeches/Meeting Papers']
- Published in
- United States of America
Table of Contents
- Ryoko Ito 2
- CTB McGraw-Hill 2
- Computerized adaptive te.sts CAT cannot be successfully 3
- An obvious way to 3
- Because items should yield parametelr estimates similar 3
- Once items were selected for 4
- PARMATE Burket 1990 a program 4
- The the full-length PP form. 4
- Study Design 4
- The CAT forms were constructed 4
- Hard Medium Difficulty and Easy. by less competent examinees. 4
- Paper-and-pencil samples would be representative 4
- The information on the forms bank b-values. 5
- The two right columns 5
- The items in the Medium The Easy CAT form Note that the CAT forms differed in 5
- -0.98 with a mean of -1.34. the standard deviation of b-values. 5
- All three CAT forms had 5
- For comparison 5
- Characteristics of the Samples 6
- CAT Samples B-Value Range 6
- -0.63 - -0.5 - 6
- -1.7 - -1.05 6
- Group. 6
- Overall N mean theta 7
- Furthermore the Ethnic Group 7
- 100 200 400 and Samples of 100 7
- For purposes 7
- Samples. 7
- Agreement 7
- Results 8
- The CAT sample. 8
- Difficulty form. 8
- The higher correlations and lower 8
- CAT of a sample include the mean ability of tle reference 9
- The 9
- Difficulty form. and 9
- The a comparable value .864. 9
- All the correlations for 10
- The better performance of the 10
- The results for the Ethnic Group As reflected in the mean theta are the largest 10
- Sirulated CAT Forms 10
- As expected the correlations and MADs CAT forms obtained using the 10
- The the 11
- However with the Hard form b- 11
- The reference group of The Ethnic 11
- Table 6. were In contrast the MADs exhibit.xl only slight 11
- These results once 11
- In comparison 12
- Namely the procedure 12
- In a similar 13
- Benchmark 13
- The results improved to less than one half a greater b-value standard 14
- However by a wider range of items.1 14
- Although this would recalibrated using 14
- Increasing the 14
- A lower starting point whose used of this approach may have 14
- For examinees from maximize 14
- References 15
- Lord F.M. 1980. 15
- Associates Inc. 15
- Mean SD Mean SD 16
- Easy 16
- Mini 16
- -3.06 16
- Table 2 17
- Subpopulations 17
- 1st-Time US 17
- Mean 17
- Mean SD 17
- Mean 17
- Mean 17
- All cases combined. 17
- Table 3 18
- Correlation Bw Bank bs and Part-form bs r 18
- Their Mean Absolute 18
- Simulated CAT Forms 18
- Sample 18
- Hard -0.63 18
- Ability Range 18
- MAD 18
- Between -1.0 and -0.5 logits. Between -0.5 and 0.0 logits. 18
- Table 4 19
- Their Mean Absolute Difference MAD 19
- CAT Sample Repre- 19
- MAD 19
- Mean 19
- Form Sample 20
- First-time U.S. 20
- Ability Range 20
- CAT Sample Repre- sentative 20
- MAD MAD 20
- Mean 20
- 200 20
- Mean 20
- Table 6 21
- Their Mean Absolute 21
- Form 21
- Item 21
- Hard -.63 - .86 21
- CAT 21
- MAD 21
- MAD MAD MAD 21
- Mean 21
- Table 7 22
- CAT Sample 22
- MAD 22
- Table 8 23
- Sample 23
- Repre- 23
- MAD 23
- Between -1.0 and -0.5. 23
- Between -0.5 and 0.0. 23
- Abstract 24
- CAT examination. 24
