Validity of CTBS from an Analysis of Speededness and Item Response Patterns
Keywords:
test validity, Sato caution index, educational measurementAbstract
This study examines the validity of the Canadian Tests of Basic Skills from the perspective that several of the subtests of this battery are highly speeded. One tool in the investigation is a modification of the Sato caution index, an index which measures the congruence of individual response patterns with group patterns. Results show that there is very little relationship between the number of items attempted and the number correct. Students who attempt the most questions have answer patterns least congruent with the group pattern, strongly suggesting that those who attempt more items do so by guessing. Slow- and fast-working students are not distributed randomly across schools, thus allowing identification of slow- and fast-working schools. This suggests that guessing behaviour may be at least partly controlled by teacher direction rather than individual preference. Implications for the validity of item difficulties and comparisons across groups and individuals and across subtests are discussed.
References
Alderman, D. L., Swinton, S. S., & Braswell, J. S. (1979). Assessing basic arithmetic skills and understanding across curricula: Computer-assisted instruction and compensatory education. The Journal of Children's Mathematical Behavior, 2, 3-28.
Baxter, B. (1941). An experimental analysis of speed and level in an intelligence test. Journal of Educational Psychology, 32, 285-296.
Beck, L. F. (1933). The role of speed and intelligence. Psychological Bulletin, 30, 169-178.
Bennett, G. K., & Doppelt, J. E. (1956). Item difficulty and speed of response. Educational and Psychological Measurement, 16, 494-496.
Birenbaum, M., & Tatsuoka, K. K. (1982). On the dimensionality of achievement test data. Journal of Educational Measurement, 19, 259-266.
Boag, A. K., & Neild, M. (1962). The influence of the time factor on the scores of the Triggs Diagnostic Reading Test as reflected in the performance of secondary school pupils. Journal of Educational Research, 55, 181-183.
Bridgeman, B. (1979). Generality of a "fast" or "slow" test-taking style across a variety of cognitive tasks. Paper presented at a joint meeting of the American Educational Research Association and the National Council on Measurement in Education, San Francisco.
Burack, B. (1967). Relationship between course examination scores and time taken to finish the examination, revisited. Psychological Reports, 20, 164.
Cronbach, L. J., & Warrington, W. G. (1951). Time limit tests: Estimating their reliability and degree of speededness. Psychometrika, 16, 167-188.
Daly, J. L., & Stahmann, R. F. (1968). The effects of time limits on a university placement test. Journal of Educational Research, 62, 103-104.
Davidson, W. M., & Carroll, J. B. (1945). Speed and level components in time limit scores: A factor analysis. Educational and Psychological Measurement, 5, 411-427.
Divgi, D. R. (1983). Some reservations concerning indices for unusual response patterns. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal.
Donlon, T. F. (1981). Uninterpretable scores: Their implication for testing practice. Journal of Educational Measurement, 18, 213-220.
Douglass, J. B. (1981). Item bias, test speededness, and Rasch tests of fit. Paper presented at the annual meeting of the American Educational Research Association, Los Angeles.
Drasgow, F. (1982). Choice of test model for appropriateness measurement. Applied Psychological Measurement, 6, 297-308.
Dubin, J. A., Osburn, H., & Winick, D. A. (1969). Speed and practice: Effects on Negro and White test performances. Journal of Applied Psychology, 53, 19-23.
Evans, F. R., & Reilly, R. P. (1972). A study of speededness as a source of test bias. Journal of Educational Measurement, 9, 123-131.
Gulliksen, H. (1950). Theory of mental tests. New York: John Wiley and Sons.
Guttman, L. (1950). The basis for scalogram analysis. In S. A. Stauffer (Ed.), Measurement and prediction. Princeton, NJ: Princeton University Press.
Harnisch, D. L. (1981). Analysis of item response patterns: Consistency indices and their application to criterion referenced tests. Paper presented at the annual meeting of the American Educational Research Association, Los Angeles.
Harnisch, D. L. (1983). Item response patterns: Applications for educational practice. Journal of Educational Measurement, 20, 191-206.
Harnisch, D. L., & Linn, R. L. (1981). Analysis of item response patterns: Questionable test data and dissimilar curriculum practices. Journal of Educational Measurement, 18, 133-146.
Himmelweit, H. T. (1946). Speed and accuracy of work as related to temperament. British Journal of Psychology, 36, 132-144.
Joreskog, K. G. (1979). Analyzing psychological data by structural analysis of covariance matrices. In J. Magidson (Ed.), Advances in Factor Analysis and Structural Equation Models. Cambridge, MA: Abt Books.
Khan, S. B. (1968). The relative magnitude of speed and power in SCAT. Journal of Educational Measurement, 5, 327-329.
Levine, M. V., & Drasgow, F. (1983). The relation between incorrect option choice and estimated ability. Educational and Psychological Measurement, 43, 675-686.
Linn, R. L., & Harnisch, D. L. (1981). Interactions between item content and group membership on achievement test items. Journal of Educational Measurement, 18, 109-118.
Lord, F. M. (1956). A study of speed factors in tests and academic grades. Psychometrika, 21, 31-50.
Maruhnich, N. (1980). An empirical comparison of four indicators of test speededness. Paper presented at the annual meeting of the American Educational Research Association, Boston.
McArthur, D. L. (1982). Detection of item bias using analyses of response patterns. Paper presented at the annual meeting of the American Educational Research Association, New York.
Mollenkopf, W. G. (1960). Time limits and the behavior of test takers. Educational and Psychological Measurement, 20, 223-230.
Moreton, C. A., & Butcher, H. J. (1963). Are rural children handicapped by the use of speeded tests in selection procedures? British Journal of Educational Psychology, 33, 22-30.
Myers, C. T. (1952). The factorial composition and validity of differently speeded tests. Psychometrika, 17, 347-352.
Newfoundland Department of Education. (1979). Newfoundland standards testing program, Grade Four, 1978: Final report. St. John's.
Petersen, N. S. (1980). Bias in the selection rule, bias in the test. In L. J. T. van der Kamp, W. F. Langerak, & D. N. M. de Gruiter (Eds.), Psychometrics for educational debates (pp. 103-122). Chichester, U.K.: John Wiley and Sons.
Rindler, S. E. (1979). Pitfalls in assessing test speededness. Journal of Educational Measurement, 16, 261-270.
Rindler, S. E. (1980). The effects of skipping over more difficult items on time-limited tests: Implications for test validity. Educational and Psychological Measurement, 40, 989-998.
Rudner, L. M. (1983). Individual assessment accuracy. Journal of Educational Measurement, 20, 207-220.
Rudner, L. M., Getson, P. R., & Knight, D. L. (1980a). Biased item detection techniques. Journal of Educational Statistics, 5, 213-234.
Rudner, L. M., Getson, P. R., & Knight, D. L. (1980b). A Monte Carlo comparison of seven biased item detection techniques. Journal of Educational Measurement, 17, 1-10.
Stafford, R. E. (1971). The speededness quotient: A new descriptive statistic for tests. Journal of Educational Measurement, 8, 275-277.
Swineford, P. (1956). Technical manual for users of test analyses (Statistical Report 56-42). Princeton, NJ: Educational Testing Service.
Tatsuoka, K. K. (1983). Rule space: An approach to dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345-354.
Tatsuoka, K. K. (1984). Caution indices based on item response theory. Psychometrika, 49, 95-110.
Tatsuoka, K. K., & Linn, R. L. (1983). Indices for detecting unusual response patterns: Links between two general approaches and potential applications. Applied Psychological Measurement, 7, 81-96.
Tatsuoka, K. K., & Tatsuoka, M. M. (1983). Spotting erroneous rules of operation by the individual consistency index. Journal of Educational Measurement, 20, 221-230.
Terranova, G. (1972). The relationship between test scores and test time. Journal of Experimental Education, 40, 81-83.
Traub, R. E., & Hambleton, R. K. (1972). The effect of scoring instructions and degree of speededness on the validity and reliability of multiple-choice tests. Educational and Psychological Measurement, 32, 737-758.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
The Canadian Journal of Education follows Creative Commons Licencing CC BY-NC-ND.
