NAEP RESEARCH AND DEVELOPMENT

The NAEP psychometric and statistical methods research team applies advanced psychometric and statistical methods to investigate a wide range of methodological and substantive issues related to measurement in large-scale assessments. Potential topics for the internship program include: (1) improving proficiency estimation through marginal maximum likelihood latent regression for NAEP data and (2) implementing missing data techniques to large-scale data analysis. Applicants with knowledge and research experience in item response theory (IRT), latent regression, plausible value methodology, multiple imputations, and analysis of complex sample data are strongly encouraged to apply. Proficiency in R programming and various IRT/missing data R packages is expected. Familiarity with programming languages such as Bayesian modeling with Stan is an asset.

Selected psychometrics & statistical methods projects from prior interns include:

• Comparing Mantel–Haenszel and Wald differential item functioning (DIF) detection methods under matrix sampling of items
• Exploring gender-related differential item time functioning in digitally based assessments
• Examining the effect of digital familiarity on writing performance using multiple-group analysis
• Designing a novel multistage testing (MST) assembly method for a test with many subscales
• Applying Bayesian region of measurement equivalence (ROME) method to examine the invariance of NAEP student questionnaire index variables

As part of the eNAEP digital assessment delivery system, process data is collected. Process data represent student interactions with the assessment platform and the assessment tasks; the data consist of time-stamped records of student actions or activities (e.g., highlighter use and answer selection) as well as automatically generated actions (e.g., switching to the next section due to time out). Process data have the potential to provide valuable insights into students’ response processes and behaviors as well as aid in evaluating how students interact with assessment items, system functions, and exam delivery platform. Process data can be used as the primary data source or the auxiliary data source on a wide range of projects.

Potential projects in foundational process data research include data privacy and quality (e.g., developing a systematic approach to create process data sets and ensure information quality), implementing methods for the extraction of useful information from process data (e.g., pattern recognition, uncertainty modeling), and exploration of new methods for analyzing process data (e.g., sequence mining and process mining). Strong statistical skills and the ability to work effectively with large and complex data as well as good programming skills (R and/or Python) are required. Experience in advanced statistical modeling/machine-learning methods are preferred.

Selected process data projects from prior interns include:

• Understanding Students’ Problem-Solving Processes via Action Sequence Analyses
• Item profiling: Insights into characteristics and design of each item
• A Social Network Analysis of Answer Change Behavior Using NAEP Process Data
• Insights from students’ computations using the calculator
• A comparison of student disengagement across years
• Whether items can be reconstructed from the process data

The policy-relevant research team uses NAEP data to explore research questions informed by education, sociology, economics, psychology, public health, and any other related disciplines that have the potential to affect policy and practice in K–12 education. Topics for the internship program will focus on: (1) text analytics of teacher responses to the write-in questions in the 2019 National Indian Education Study (NIES) to identify effective teaching and learning strategies for American Indian and Alaska Native students as well as critical sociocultural contexts of their school and community, (2) investigating green space around NAEP Trial Urban District Assessment (TUDA) schools and its relationship with student performance, and (3) examining state-level variations in course offerings and their relationship to student achievement. Applicants with an interest in any of the contexts addressed above as well as experience with advanced statistical modeling methods (e.g., topic modeling, text categorization, geospatial analysis, multi-level modeling) are strongly encouraged to apply. Familiarity with R, desktop GIS, Stata, or Mplus is preferred.

Selected projects in policy-relevant research projects from prior interns include:

• Investigating the relationship between air pollution and education outcomes using NAEP and county-level data
• The relationship between algebra skills, NAEP performance, and postsecondary outcomes
• Behavioral incidents, school behavioral climate, and student level mathematics achievement: Exploration of NAEP and the School Survey on Crime and Safety (SSOCS)
• The relationships between student experiences and NAEP technology and engineering literacy achievement
• Examining the relationship between mathematics growth and growth in working memory in the early grades using the Early Childhood Longitudinal Study, Kindergarten Class of 2010–11 (ECLS-K:2011)

The project in this topic area will focus on expanding the nature and scope of reports based on NAEP data by combining communication and information science skills for highly-engaged and technically sophisticated constituencies, such as NAEP’s National Assessment Governing Board (NAGB) members, assessment experts, and other stakeholders. The intern will be part of a multidisciplinary team composed of NCES and AIR staff, including analysts, graphic designers, measurement scientists, and programmers.

The intern will use demographic and performance data from NAEP and other federal datasets to explore aspects of student learning experiences. Topics for the internship program will include: (1) an examination of instruction modes (e.g., remote, hybrid), NAEP performance by student groups (e.g., region, location), and the impact of COVID-19 (e.g., cases, deaths); and (2) an exploration of different aspects of school climate (e.g., school disruptions like school violence), learning experiences (e.g., hours of live teacher instruction), and variation in NAEP performance. The ideal applicant will possess an interest in communicating statistical information; experience creating advanced, interactive data visualizations; and demonstrable technical proficiency in their chosen programming framework.

Selected projects from prior interns in this area include:

• Developing interactive data visualizations that accurately convey statistically valid insights drawn from combining NAEP with other federal and state data sets.
• Developing interactive visualizations allowing the comparison across states and time of student access to high-speed internet and computer resources over the last decade.