North Carolina (Lead State), Kentucky, New Mexico, and South Carolina’s Online Mathematics Assessment Project Documents the Reliability and Technical Quality
of Technology-Enhanced Items and Items Scored Using Artificial Intelligence
Pacific Metrics Corporation, a leading provider of online assessment solutions, announces the completion of an Enhanced Assessment Grant (EAG) project funded by the U.S. Department of Education. The Online Mathematics Assessment Project (OMAP) was a collaborative effort by four state education agencies (North Carolina, Kentucky, New Mexico, and South Carolina) and Pacific Metrics Corporation to investigate the feasibility of assessing student test results in mathematics online using test item formats other than multiple choice.
Started in late 2010, OMAP studied the feasibility of using online assessments that consist entirely of constructed-response and technology-enhanced test items in two courses, grade 7 mathematics and Algebra I, and the reliability and validity of automated scoring. The results of the OMAP study were released on October 31, 2012 and are available at /omap. They show that alternative item types including constructed-response and technology-enhanced items, delivered online and scored using CRASETM, Pacific Metrics’ automated scoring solution, are a reliable way to assess a student’s knowledge of mathematics.
Pacific Metrics and the partner states developed a computer-based, online assessment that was delivered in spring 2012 to over 9,000 students in the four states. Results were evaluated to compare the technical quality of tests consisting of constructed-response and technology-enhanced items to traditional multiple-choice assessments. The assessments allowed for automated scoring of the items and incorporated accommodations to meet the needs of English language learners and students with disabilities. The consortium of states and Pacific Metrics’ skilled assessment team worked together with a panel consisting of nationally recognized experts in mathematics education and in the education of English learners and students with disabilities. The project was discussed at the National Conference on Student Assessment in Minneapolis in June 2012 under the title “Using Technology for Better Measurement: An Evaluation of Machine-Scored Online Constructed-Response Assessments in Mathematics.”
The items were aligned to the Common Core State Standards. In addition to conventional multiple-choice items, the following item types were included:
- Graphing: allows students to draw lines or points (or both) on a grid
- Drag-and-Drop: enables students to match images to their correct counterpart or sort and order components of a response
- Constrained Response: provides students with an input tool that pops up when they click in the answer box; students can use either the tool buttons or equivalent keystrokes to enter the response
- Table Entry: enables students to use the constrained-response tool to enter responses in a table
- Free Text Entry: removes restrictions on what students can enter into the answer box; the entire keyboard is available
- Mixed: incorporates both constrained response and free text entry response in different parts of the item
“This study demonstrated that online, automatically scored constructed-response and technology-enhanced mathematics items can be administered in a variety of settings and yield reliable results,” says Dr. Phoebe Winter, OMAP Project Manager and Executive Vice President of Education Policy at Pacific Metrics. “The administration can be successfully standardized, and students with disabilities and English learners can be accommodated via online or in-person accommodations. The findings will be helpful to state departments of education and the assessment consortia as they design their next-generation assessment systems.”
For more information about the Online Mathematics Assessment Project, visit /omap.
About Pacific Metrics
Pacific Metrics Corporation creates lasting improvements to the assessment and learning environment through the thoughtful use of technology. Headquartered in Monterey, California, Pacific Metrics provides specialized psychometric software and sophisticated online systems to state departments of education and to organizations that develop, administer, and score large-scale assessments. The company has achieved recognition for its technical work and for being a leading force in the development of innovative, customized, web-based systems. For more information, visit www.pacificmetrics.com.