Awarded Cooperative Agreements & Special Studies

Awardee: Measurement Inc.

NAEP reading and mathematics assessments have been scored by human readers since the introduction of such items into the program. Advances in automated scoring have reached the point that scoring responses to content-specific constructed-response items using an automated scoring engine may produce benefits over hand scoring. To investigate this possibility, a study using hand-scored responses to recently administered NAEP reading and mathematics items is being conducted by Measurement Inc. The goals of the study are to (1) determine whether an automated scoring engine can accurately predict the scores assigned by human readers, (2) compare the costs and benefits of automated scoring vs. human scoring, and (3) establish a protocol for future automated scoring research.

The investigators will employ the automated scoring engine Project Essay Grade (PEG®) to create and validate scoring models for reading and mathematics items. Specifically, they will draw samples of student responses in both content areas from recent administrations of NAEP. Using these samples, the investigators will construct models to replicate scores assigned by human readers and cross-validate them by applying them to independent samples of student responses. The investigators will compare the scores assigned by PEG with the scores assigned by human readers, reporting results in terms of raw score agreement rates, correlation coefficients, and kappa coefficients. Finally, a cost-benefit analysis of automated scoring will be conducted that compares not only the reliability of automated scores to that of human scorers but also the cost of deriving the scores.

When they have completed the study, the investigators will work with NAEP staff and officials to map out a program of research in which to apply automated scoring to the remaining content areas.

How this work supports NAEP

The automatic scoring research study addresses Phase 6 (Score the Assessment) of the NAEP assessment process. As part of the NAEP R&D program’s cooperative agreement opportunity, this research study meets the strategic vision to Inform, Innovate, & Engage by strengthening and expanding “partnerships by broadening stakeholders’ awareness of NAEP and facilitating their use of NAEP resources.”

Awardee: ACT, Inc.

NAEP is the only comparative assessment of academic competencies regularly administered to nationally representative samples of students enrolled in grades 4, 8, and 12. However, due to a low-stakes assessment context in NAEP, there are long-standing questions about the level of engagement and effort of the participating students and, consequently, about the validity of the reported results. For assessment results to be valid, students should be sufficiently engaged with the assessment tasks to reflect their knowledge and skills accurately. The current project, which has been undertaken by ACT, Inc., investigates the effects of engagement-enhanced features (EEFs) on the performance of 8th-graders on digital-based assessments (DBA). The goal of implementing EEFs is to help reduce the likelihood that assessment scores underestimate students’ knowledge and skills. Specifically, this project aims to identify problematic items in current NAEP DBA items based on analyses of process data and empirically explore ways to increase student engagement in these items by adding EEFs. Four classes of EEFs will be developed for selected items, focusing on the following engagement facets: relevance, authenticity, cognitive complexity, and self-assessment. The EEF and original NAEP items will be tested and validated with students through cognitive labs using multimodal measures of student engagement.

The ACT project team will cooperate with NCES project staff on all activities, including prioritization of NAEP domains and item types to be included in the project; identifying data from the most recent NAEP assessments; development and validation of the EEFs through cognitive labs with students; overall analysis and reporting; and collaborative work on providing operational recommendations for further research and development in NAEP.

How this work supports NAEP

The student engagement research study addresses Phases 2 (Create the Assessment), 5 (Administer the Assessment), 6 (Score the Assessment), and 7 (Analyze and Report the results) of the NAEP Assessment Process. As part of the NAEP R&D program’s cooperative agreement opportunity, this research study meets the strategic vision to Inform, Innovate, & Engage by strengthening and expanding “partnerships by broadening stakeholders’ awareness of NAEP and facilitating their use of NAEP resources.”

Awardee: Dr. Igor Griva

An accurate assessment of educational progress remains a challenging task mainly because students have various levels of preparedness as well as individual learning and expression styles. For example, some students may score better in a test when they are given a large number of multiple-choice items while other students are just the opposite: they can make excellent and thorough answers to constructed response items, but are unable to quickly switch attention among a large number of simple testing problems. Therefore, when testing students’ knowledge, an exam must have an adequate number of problems to cover various subject areas, difficulty levels, and question formats. However, in the absence of a systematic and scientific approach to test building, exams often suffer from an inability to reconcile multiple goals and criteria. They may focus excessively on testing some subject areas while only scarcely covering others. Or they may be too focused on one particular format or be too difficult or simple.

The goal of the current project is to improve the quality of tests capable of accurate assessment of student knowledge by providing scientifically sound methodology and software for test development. The methodology being used is based on optimization algorithm, which provides the best possible examination design based on specified goals, constraints, and psychometric data. The methodology will be implemented through a novel, user-friendly software that places mathematically challenging modeling and implementation issues at the "back end" and provides a user-friendly interface, or “front end,” that will control the specifications of the tests that need to be built.

As a result of this effort, the education community will have an easy-to-use tool, TestDesign, capable of assisting specialists in the preparation of high-quality tests to advance the goals of the National Assessment of Educational Progress.

How this work supports NAEP

The TestDesign methodology and software study address Phase 2 (Create the Assessment) of the NAEP Assessment process. As part of the NAEP R&D program’s cooperative agreement opportunity, this research study meets the strategic vision to Inform, Innovate, & Engage by strengthening and expanding “partnerships by broadening stakeholders’ awareness of NAEP and facilitating their use of NAEP resources.”

Awardee: Dr. David Kaplan

Of critical importance to education policy is monitoring trends in education outcomes over time. In the United States, NAEP has provided data since 1969, with trend data since 1990 at the national level and since 1992 at the state level. In addition, since 2002, selected urban districts (on a trial basis) have also participated. Thus, NAEP can provide important monitoring and forecasting information regarding population-level academic performance. The purpose of this study is to provide a “proof-of-concept” that state NAEP assessments can be used to (1) specify cross-state growth regressions; and (2) develop Bayesian probabilistic predictive models that can be used to forecast trends across states in important educational outcomes while accounting for uncertainty in every step of the modeling process.

How this work supports NAEP

The Bayesian probabilistic forecasting with state NAEP data study addresses Phase 7 (Analyze and Report the Results) of the NAEP Assessment Process. As part of the NAEP R&D program’s cooperative agreement opportunity, this research study meets the strategic vision to Inform, Innovate, & Engage by increasing opportunities to connect NAEP to administrative data and state, national, and international student assessments as well as improving the analysis and reporting of NAEP contextual variables by considering potential to provide meaningful context and insights for policy and practice.