The Online Mathematics Assessment Project
The Online Mathematics Assessment Project (OMAP) was a collaborative effort by four states and Pacific Metrics Corporation to investigate the properties and feasibility of assessing student mathematics using item formats other than multiple choice (MC). The departments of education in the four states, North Carolina (the lead state), Kentucky, New Mexico, and South Carolina, received a Grant for Enhanced Assessment Instruments from the U.S. Department of Education and began the project in late 2010.
The study compared how online mathematics tests consisting of constructed-response (CR) and technology-enhanced (TE) items compared to tests consisting of MC items, in two courses, grade 7 mathematics and Algebra I. The CR and TE item responses were all automatically scored by computer after the scoring engine was “trained” using pilot test data. Researchers addressed four broad research questions:
- Can computer-delivered CR/TE mathematics items measure essential knowledge, skills, andprocesses in mathematics content standards more closely than traditional MC items?
- What computer-based accommodations are appropriate and feasible to meet a broad range ofstudent needs, and do they increase student access to test content?
- Does automated scoring provide reliable, valid, and acceptable (to users) item scores, and do these qualities apply equally to responses from different student groups, particularly English learners and students with disabilities?
- Does the application of measurement models associated with CR items (e.g., generalized partial-credit models) reduce error in test scores through (a) the elimination of the need to estimate the guessing parameter (associated with selected-response items) and (b) the ability to score for partial knowledge?
The results of the study demonstrate that online, automatically scored CR and TE mathematics item scan be administered in a variety of settings and yield reliable results; this is especially true of items with well-defined or constrained-response requirements. The administration can be successfully standardized, and most students with disabilities and students who are English learners can be accommodated via online or in-person accommodations. The study’s focus on a single subject area (mathematics), two grade levels (grade 7 and Algebra I), and a limited number of item types prevents broad generalizations, but it does provide a proof of concept supporting the use of such items in large-scale assessment.
Use the links on the left to learn more about the project.
This work was supported by a Grant for Enhanced Assessment Instruments awarded by the U.S. Department of Education to the North Carolina Department of Public Instruction as lead state, award S368A100003. The opinions in documents produced as part of the grant are those of the authors and do not necessarily reflect those of the North Carolina Department of Public Instruction, the U.S. Department of Education, or the participating project states.