Skip to Content

Computerized Multistage Testing

Theory and Applications

Edited by Duanli Yan, Alina A. von Davier, Charles Lewis

Chapman and Hall/CRC – 2014 – 546 pages

Series: Chapman & Hall/CRC Statistics in the Social and Behavioral Sciences

Purchasing Options:

  • Add to CartHardback: $89.95
    978-1-46-650577-3
    May 6th 2014

Description

Devising tests that evaluate a nation’s educational standing and implement efficacious educational reforms requires a careful balance among the contributions of technology, psychometrics, test design, and the learning sciences. Unlike other forms of adaptive testing, multistage testing (MST) is highly suitable for testing educational achievement because it can be adapted to educational surveys and student testing. Computerized Multistage Testing: Theory and Applications covers the methodologies, underlying technology, and implementation aspects of this type of test design.

The book discusses current scientific perspectives and practical considerations for each step involved in setting up an MST program. It covers the history of MST, test design and implementation for various purposes, item pool development and maintenance, IRT-based and classical test theory-based methodologies for test assembly, routing and scoring, equating, test security, and existing software. It also explores current research, existing operational programs, and innovative future assessments using MST.

Intended for psychologists, social scientists, and educational measurement scientists, this volume provides the first unified source of information on the design, psychometrics, implementation, and operational use of MST. It shows how to apply theoretical statistical tools to testing in novel and useful ways. It also explains how to explicitly tie the assumptions made by each model to observable (or at least inferable) data conditions.

Contents

Test Design, Item Pool, and Maintenance

Overview of Computerized Multistage Tests Duanli Yan, Charles Lewis, and Alina A. von Davier

Linear Tests and Computerized Adaptive Tests (CATs)

Multistage Tests (MSTs)

MST Designs for Different Purposes

Implementation Schemes

Designing MST

Content Balance and Assembly

Exposure Control

Routing

Scoring, Linking, and Equating

Reliability, Validity, Fairness, and Test Security

Current and Future Applications

Logistic Challenges

Summary

Multistage Test Designs: Moving Research Results into Practice April L. Zenisky and Ronald K. Hambleton

The MST Design Structure

The State of Research: MST Development and Design Considerations

Conclusions and Next Steps

Item Pool Design and Maintenance for Multistage Testing Bernard P. Veldkamp

Designing an Item Pool Blueprint

Applications in Item Writing

Maintenance

Discussion

Mixed-Format Tests: Issues and Methods Jiseon Kim and Barbara G. Dodd

Literature Review on Design Components in Mixed-Format MST

Comparing Other Testing Approaches

Issues and Future Research Suggestions for Mixed-Format MST

Conclusion

Design and Implementation of Large-Scale Multistage Testing Systems Richard Luecht

MST Design and Implementation Considerations

Conclusions: A Research Agenda

Test Assembly

Overview of Test Assembly Methods in Multistage Testing Yi Zheng, Chun Wang, Michael J. Culbertson, and Hua-Hua Chang

MST Framework

MST Assembly Design

Automated Assembly for MST

Setting Difficulty Anchors and Information Targets for Modules

"On-the-Fly" MST (OMST) Assembly Paradigm

MST, CAT, and Other Designs—Which Way to Go?

Using a Universal Shadow-Test Assembler with Multistage Testing Wim J. van der Linden and Qi Diao

Solving Shadow-Test Assembly Problems

Basic Design Parameters

Different Testing Formats

Relative Efficiency of Formats

Empirical Study

Concluding Comments

Multistage Testing by Shaping Modules on the Fly Kyung (Chris) T. Han and Fanmin Guo

MST by Shaping

MST-S versus MST-R versus CAT

Discussion and Conclusion

Optimizing the Test Assembly and Routing for Multistage Testing Angela Verschoor and Theo Eggen

Optimizing MST Assembly: A Nonexhaustive Search

Limited Item Pools, Two- and Three-Parameter Models

Discussion

Routing, Scoring, and Equating

IRT-Based Multistage Testing Alexander Weissman

Introduction

Motivation for Tailored Testing

Routing Rules

Scoring and Classification Methodologies

Final Comments

A Tree-Based Approach for Multistage Testing Duanli Yan, Charles Lewis, and Alina A. von Davier

Regression Trees

Tree-Based Computerized Adaptive Tests

Tree-Based Multistage Testing

Algorithm

An Application

Discussion

Limitations and Future Research

Multistage Testing for Categorical Decisions Robert Smith and Charles Lewis

Computer-Mastery Methods

Information Targeted at Cut versus at Ability

Influence of Multiple Cut Scores

Factors That Can Reduce Optimal Solutions

Example Based on Smith and Lewis (1995)

Adaptive Mastery Multistage Testing Using a Multidimensional IRT Model C.A.W. Glas

Introduction

Definition of the Decision Problem

Computation of Expected Loss and Risk Using Backward Induction

Selection of Items and Testlets

Simulation Studies

Conclusions and Further Research

Multistage Testing Using Diagnostic Models Matthias von Davier and Ying (Alison) Cheng

The DINA Model and the General Diagnostic Model

Experience with CD-CATs

CD-MSTs

Discussion

Considerations on Parameter Estimation, Scoring, and Linking in Multistage Testing Shelby J. Haberman and Alina A. von Davier

Notation

The Item Response Model

The Test Score

Approaches to Parameter Estimation

Conclusions

Test Reliability, Validity, Fairness, and Security

Reliability of Multistage Tests Using Item Response Theory Peter W. van Rijn

Test Reliability

Application: IRT Reliability for MST in NAEP

Conclusion

Multistage Test Reliability Estimated via Classical Test Theory Samuel A. Livingston and Sooyeon Kim

The Estimation Procedure

Testing the Accuracy of the Estimation Procedure

How Accurate Were the Estimates?

Evaluating Validity, Fairness, and Differential Item Functioning in Multistage Testing Rebecca Zwick and Brent Bridgeman

Content Balancing

Opportunities for Item Review and Answer Changing

Skipping Strategies

MST Routing Algorithms

The Digital Divide

Comparability of Computer Platforms

Accommodations for Students with Disabilities and English Language Learners

Differential Item Functioning Analysis in MSTs

Application of the Empirical Bayes DIF Approach to Simulated MST Data

Summary

Test Security and Quality Control for Multistage Tests Yi-Hsuan Lee, Charles Lewis, and Alina A. von Davier

An Overview of a Three-Component Procedure

Tools to Evaluate Test Security and Quality Control

A Simulation Study Using CUSUM Statistics to Monitor Item Performance

Discussion

Applications in Large-Scale Assessments

Multistage Test Design and Scoring with Small Samples Duanli Yan, Charles Lewis, and Alina A. von Davier

Small Data Sample

Item Pool

Various MST Module Designs

Routing and Scoring

Comparisons of the Six MST Designs

Discussion

Limitations and Future Research

The Multistage Test Implementation of the GRE Revised General Test Frédéric Robin, Manfred Steffen, and Longjuan Liang

From CAT to MST

MST Design

Implementation

Monitoring

Summary

The Multistage Testing Approach to the AICPA Uniform Certified Public Accounting Examinations Krista J. Breithaupt, Oliver Y. Zhang, and Donovan R. Hare

Research on Multistage Testing

Item Bank Development for MST

Content Security Monitoring for MST

Inventory Exposure Planning for MST

Discussion

Transitioning a K–12 Assessment from Linear to Multistage Tests Carolyn Wentzel, Christine M. Mills, and Kevin C. Meara

Administering CTP Items Online

Creating a New MST Scale Using IRT

Multistage-Adaptive Test Development

Score Reporting

Summary

A Multistage Testing Approach to Group-Score Assessments Andreas Oranje, John Mazzeo, Xueli Xu, and Edward Kulick

Targeted Testing

Goals of the Study

Methods

Results

Discussion

Controlling Multistage Testing Exposure Rates in International Large-Scale Assessments Haiwen Chen, Kentaro Yamamoto, and Matthias von Davier

Item Exposure Rate Control for Multistage Adaptive Assessments

Method: How to Compute and Adjust the Item Exposure Rates

Data

Results

Conclusion

Software Tools for Multistage Testing Simulations Kyung (Chris) T. Han and Michal Kosinski

MSTGen

R

Conclusions

Closing Remarks

Past and Future of Multistage Testing in Educational Reform Isaac I. Bejar

Future of MST

A Model-Based Three-Stage Design

Item Generation and Automated Scoring and Broadly Accessible Test Content

Summary and Conclusions

Bibliography

Name: Computerized Multistage Testing: Theory and Applications (eBook)Chapman and Hall/CRC 
Description: Edited by Duanli Yan, Alina A. von Davier, Charles Lewis. Devising tests that evaluate a nation’s educational standing and implement efficacious educational reforms requires a careful balance among the contributions of technology, psychometrics, test design, and the learning sciences. Unlike other...
Categories: Regression Analysis and Multivariate Statistics