2021 saw work featured by the NAEP R&D Program make noteworthy progress and impacts on multiple fronts. As we end this year, let’s take a look at some of those achievements.

Process Data

• In March, process data research by Emmanuel Sikali (NCES), Juanita Hicks (AIR), and Ruhan Circi (AIR) was featured on the cover of Education Measurement: Issues and Practice (Vol 40, Issue 1, Spring 2021). The article accompanying the cover noted the value of using process data to inform timing assumptions for different item types in standardized assessments.

Figure 1. Cover featuring Where Data Meet Assumptions: Visualization of Multiple-Choice Item Response Time. From “On the Cover: Time Spent on Multiple-Choice Items” by Z. Cui, 2021. Educational Measurement Issues and Practice, Volume 40 (Issue 1).  Retrieved from https://onlinelibrary.wiley.com/doi/10.1111/emip.12420

• In May, the Center for Process Data released two new process data visualizations reflecting what process data can reveal in large-scale assessments.
• In July, the Process Data team won the first IES Process Data related grant: Rethinking Accessibility using NAEP Process Data: Exploring Universal Design Features and Accommodations (R324P210002). This grant funds a study that leverages process data, student performance data, and contextual data from 2017 NAEP Grade 8 Mathematics Assessment to investigate how students with disabilities who could take the assessment made use of accommodations and universal design elements provided in the digitally based NAEP assessment.   
• In August, the NAEP R&D Program hosted the Summer Process Data Training Series,  in which selected participants learned the basics of process data and how they relate to NAEP in particular; process data cleaning, manipulation, and variable derivation; and the application of the variable derived from process data towards addressing substantive research questions. One participant said the following: “Thank you very much for this amazing training.”
• Over the course of the year, the process data team achieved a 100% acceptance rate at major conferences, including at Educational Data Mining, Society for Research on Educational Effectiveness (SREE), National Council on Measurement in Education (NCME), and Association of Test Publishers (ATP).

EdSurvey

• In April, a new EdSurvey package was released, now including TIMSS 2019, OECD PISA 2018 Financial Literacy, and 2017 US PIACC data. In July, EdSurvey was further updated to include NHES and SOCCS data. By December, EdSurvey and its ancillary packages (i.e., WeMix and wCorr) reached 100K + downloads from R CRAN.
• In April, the EdSurvey team rolled out two new R packages: “Dire” and “NAEPirtparams.” They provide functionality for the direct estimation of NAEP data. More specifically, Dire is intended to fit the same models as the existing software AM.
• In September, EdSurvey joined the official NAEP Database Products, and NCES included EdSurvey in the NAEP Data Companion, starting with the 2019 NAEP Reading and Mathematics grades 4 and 8 assessments.
• In June, AERA’s Virtual Learning Series featured a course on using EdSurvey to analyze national and international large-scale assessment data with complex survey and matrix sampling design. Participants learned how to perform data manipulation, descriptive statistics, cross tabulations and plausible value means, and more.
• In October, the 2021 EdSurvey Large-Scale Data Analysis Training Series took place over the course of four virtual lessons. The 30 training participants received hands-on experience using the R package EdSurvey to analyze public-use NAEP and TIMSS data.
• The EdSurvey team created distance-learning modules for the NCES DLDT training site on how to analyze various NCES data, including NAEP, TIMSS, and ECLS-K: 2011. The module is scheduled to be released to the public in 2022.

NAEP Doctoral Students Internship Program

• Starting in June, seven doctoral students participated in the 2021 NAEP Doctoral Students Internship Program over the course of 10 weeks. In this highly competitive program, applicants apply to work on projects within research topic areas such as psychometrics, policy-relevant research, process data, re-envisioning quantitative information (data visualization), applying new methodological approaches to NAEP data,  or using NAEP data in novel ways. The participants received trainings and hands-on support for their research, as well as opportunities to present at major conferences and build their professional networks.

Other

• In March, NAEP researchers Yifan Bai, Markus Broer, and Stephanie Straus published their study: U.S. National and State Trends in Educational Inequality Due to Socioeconomic Status: Evidence from the 2003-17 NAEP. AIR-NAEP Working Paper 2021-01. This research used NAEP data to examine national and state trends in education inequality in the context of socioeconomic status.
• In November, the NAEP R&D Hub was launched with the goal of serving as a community center for the NAEP R&D Program. This blog shares updates on all things NAEP R&D, from announcing opportunities like new trainings and internships to featuring new and exciting secondary research using NAEP data.

In upcoming blog posts, we will share some of the ideas and innovations percolating up from our EdSurvey and Process Data teams. In the meantime, the 2022 NAEP Doctoral Internship Program is accepting applications until January 10, 2022.