NAEP Doctoral Student Internship Program

The NAEP psychometric and statistical methods research team applies advanced psychometric and statistical methods to investigate a wide range of methodological and substantive issues related to psychometrics and measurement in large-scale assessments.

One potential project topic may be evaluating measurement invariance of scales or assessments on large-scale data with complex design. Strong candidates for this topic area will have experience in item response theory (IRT), confirmatory factor analysis (CFA), plausible value methodology, differential item functioning (DIF), and analysis of complex sample data. Proficiency in R programming and Mplus is required, and experience or interest in Bayesian modeling is highly desirable. Knowledge and experience in the analytic application of intersectionality to evaluate fairness and equity in assessments is a plus.

Selected psychometric and statistical methods projects from prior interns include:

• Handling missing contextual data in large-scale assessments: The multiple imputation strategy 
• Improving proficiency estimation through marginal maximum likelihood latent regression for NAEP data
• Comparing Mantel–Haenszel and Wald differential item functioning (DIF) detection methods under matrix sampling of items
• Exploring gender-related differential item time functioning in digitally based assessments
• Examining the effect of digital familiarity on writing performance using multiple-group analysis
• Designing a novel multistage testing (MST) assembly method for a test with many subscales
• Applying Bayesian region of measurement equivalence (ROME) method to examine the invariance of NAEP student questionnaire index variables

As part of the eNAEP digital assessment delivery system, process data are collected. Process data represent student interactions with the assessment platform and assessment tasks; the data consist of time-stamped records of student actions or activities (e.g., highlighter use and answer selection) as well as automatically generated actions (e.g., switching to the next section due to time out). Process data have the potential to provide valuable insights into students’ response processes and behaviors as well as aid in evaluating how students interact with assessment items, system functions, and the exam delivery platform. Process data can be used as the primary data source or the auxiliary data source on a wide range of projects.

One potential research topic may involve investigating student writing behaviors in open-ended items. Writing aptitude and behaviors can be captured via certain linguistic features, such as students’ language use, mechanics, lexical choices, style, etc. Process data provide an interesting overview into how students build their responses. Mining process data for such features would allow us to better investigate student writing behaviors and their relationship with performance.

Methods of analysis will include natural language processing (NLP) and other computational linguistic methods. We expect the candidate to have strong statistical skills, strong programming skills in Python and/or R, some knowledge of linguistics, and the ability to work effectively with large-scale and complex data.

Selected process data projects from prior interns include:

• Analyzing response changes in constructed response items in NAEP mathematics grade 8 2022 process data 
• Understanding students’ problem-solving processes via action sequence analyses
• Item profiling: Insights into characteristics and design of each item
• A social network analysis of answer change behavior using NAEP process data
• Insights from students’ computations using the calculator
• A comparison of student disengagement across years
• Exploration of whether items can be reconstructed from the process data

The policy-relevant research team leverages NAEP data to generate valuable insights and evidence that can inform education policies at district, state, and national levels. We explore diverse topics through research questions grounded in education, sociology, economics, school finance, psychology, public health, and any other related disciplines that have the potential to shape and guide policy discussions and practices in K–12 education. 

One potential research area involves examining the relationship between teacher absenteeism and NAEP grades 4 and 8 reading/mathematics performance, with a focus on how this relationship is mediated and/or moderated by factors such as school climate, teacher qualifications, and the demographic characteristics of both the student body and teachers. The goal would be to provide actionable policy insights aimed at reducing teacher absenteeism and its impact on student outcomes. 

Applicants with interest and content knowledge in topics related to teacher absenteeism and retention as well as experience with advanced statistical modeling such as multilevel structural equation modeling are strongly encouraged to apply. Familiarity with Stata or Mplus is preferred. 

Selected policy-relevant research projects from prior interns include:

• Estimating differential course difficulty in high school mathematics courses using NAEP as a common benchmark 
• Projected and actualized post-COVID public school enrollments 
• Text analytics using the 2019 NIES teacher responses to the write-in questions 
• Investigating the relationship between air pollution and education outcomes using NAEP and county-level data 
• The relationship between algebra skills, NAEP performance, and postsecondary outcomes 
• Behavioral incidents, school behavioral climate, and student level mathematics achievement: Exploration of NAEP and the School Survey on Crime and Safety (SSOCS) 
• The relationships between student experiences and NAEP technology and engineering literacy achievement 
• Examining the relationship between mathematics growth and growth in working memory in the early grades using the Early Childhood Longitudinal Study, Kindergarten Class of 2010–11 (ECLS-K:2011) 

The re-envisioning quantitative information team focuses on expanding the nature and scope of reports based on NAEP data. The intern will be part of a multidisciplinary team composed of analysts, graphic designers, measurement scientists, and programmers. Projects in this topic area require interns to leverage their communication and information science skills to develop engaging and accessible reports for various audiences, such as educators, parents, policymakers, and researchers. 

The intern will enhance reporting capabilities by expanding output options for an R statistical package specifically designed for handling large-scale education data like NAEP’s. One potential topic may involve using the statistical package to propose ways to incorporate plot functionality and footnotes, enabling users to create graphs, charts, maps, and other desired output options. 

The ideal applicant will possess an interest in communicating statistical information; experience creating advanced, interactive data visualizations; and technical proficiency in R.

Selected projects from prior interns in this area include:

• Mathematics learning environments and the coronavirus pandemic: Relationship across locale and key equity indicators 
• Exploring equity in the collection and reporting of data
• Developing interactive data stories that accurately convey statistically valid insights drawn from combining NAEP with other federal and state data sets.
• Developing interactive visualizations allowing the comparison across states and time of student access to high-speed internet and computer resources over the last decade.

Digitally based assessments furnish abundant opportunities for the use of AI tools and methods to advance students’ test-taking experiences. They also hold the potential to support and transform various elements of the assessment cycle, including item generation, item scoring, reporting advancement, and more. 

One possible topic may involve using NAEP process data to explore the potential of generative AI and NLP methods for generating natural language descriptions of student problem-solving approaches at the level of individual student–item combinations. By measuring the alignment of generated descriptions with expert descriptions, a case can be made for the use of AI methods in identifying student strategies and misconceptions, providing a foundation for understanding problem-solving approaches and knowledge gaps across diverse student groups. 

This newest topic area is looking for applicants with a solid foundation in machine learning, data analysis, and ethical AI principles to explore the innovative use of artificial intelligence in NAEP assessments within this digital landscape. As part of our dynamic team, you will have the unique opportunity to harness AI to enhance learning outcomes and contribute to educational measurement by driving insights into student learning and problem-solving. 

Selected projects from prior interns in this area include:

• Can large language models transform automated scoring? A case study with NAEP mathematics constructed response items