NAEP Doctoral Student Internship Program

The NAEP psychometric and statistical methods research team applies advanced psychometric and statistical methods to investigate a wide range of methodological and substantive issues related to measurement in large-scale assessments.

One potential topic may be implementing missing data techniques to large-scale data analysis. Applicants with knowledge and research experience in item response theory (IRT), multiple imputations, machine learning methods (e.g., random forest, LASSO), plausible value methodology, analysis of complex sample data, and latent regression are strongly encouraged to apply. Proficiency in R programming and various IRT/missing data R packages is expected. Familiarity with programming languages such as Bayesian modeling with Stan is an asset.

Selected psychometrics and statistical methods projects from prior interns include:

• Improving proficiency estimation through marginal maximum likelihood latent regression for NAEP data
• Comparing Mantel–Haenszel and Wald differential item functioning (DIF) detection methods under matrix sampling of items
• Exploring gender-related differential item time functioning in digitally based assessments
• Examining the effect of digital familiarity on writing performance using multiple-group analysis
• Designing a novel multistage testing (MST) assembly method for a test with many subscales
• Applying Bayesian region of measurement equivalence (ROME) method to examine the invariance of NAEP student questionnaire index variables

As part of the eNAEP digital assessment delivery system, process data is collected. Process data represent student interactions with the assessment platform and the assessment tasks; the data consist of time-stamped records of student actions or activities (e.g., highlighter use and answer selection) as well as automatically generated actions (e.g., switching to the next section due to time out). Process data have the potential to provide valuable insights into students’ response processes and behaviors as well as aid in evaluating how students interact with assessment items, system functions, and exam delivery platform. Process data can be used as the primary data source or the auxiliary data source on a wide range of projects.

One potential topic may be an exploration of response change analysis for open-ended items and whether there is any indication in process data as to how and why students change their responses on open ended items. Potential methods of analysis include traditional Natural Language Processing (NLP) and novel transformer methods. We expect the candidate to have strong statistical skills, strong programming skills in Python and/or R, and the ability to work effectively with large-scale and complex data.

Selected process data projects from prior interns include:

• Understanding students’ problem-solving processes via action sequence analyses
• Item profiling: Insights into characteristics and design of each item
• A social network analysis of answer change behavior using NAEP process data
• Insights from students’ computations using the calculator
• A comparison of student disengagement across years
• Exploration of whether items can be reconstructed from the process data

The policy-relevant research team uses NAEP data to provide valuable insights and evidence that can inform education policies at district, state, and national levels. We explore diverse topics through research questions informed by education, sociology, economics, school finance, psychology, public health, and any other related disciplines that have the potential to affect policy and practice in K–12 education.

Potential topics may include: (1) measuring differential grading effects among high school mathematics courses using NAEP as a common benchmark, and (2) examining the relationship between different types of education expenditures paid from the COVID-19 Federal Assistance Fund and NAEP grade 8 mathematics performance. Applicants with an interest in any of the contexts addressed above as well as experience with advanced statistical modeling methods (e.g., structural equation modeling with polytomous categorical indicators, econometrics, multi-level modeling) are strongly encouraged to apply. Familiarity with R, Stata, or Mplus is preferred.

Selected policy-relevant research projects from prior interns include:

• Text analytics using the 2019 NIES teacher responses to the write-in questions
• Investigating the relationship between air pollution and education outcomes using NAEP and county-level data
• The relationship between algebra skills, NAEP performance, and postsecondary outcomes
• Behavioral incidents, school behavioral climate, and student level mathematics achievement: Exploration of NAEP and the School Survey on Crime and Safety (SSOCS)
• The relationships between student experiences and NAEP technology and engineering literacy achievement
• Examining the relationship between mathematics growth and growth in working memory in the early grades using the Early Childhood Longitudinal Study, Kindergarten Class of 2010–11 (ECLS-K:2011)

The re-envisioning quantitative information team focuses on expanding the nature and scope of reports based on NAEP data. The intern will be part of a multidisciplinary team composed of analysts, graphic designers, measurement scientists, and programmers. Projects in this topic area require interns to leverage their communication and information science skills to develop engaging and accessible reports for various audiences, such as educators, parents, policymakers, and researchers.

The intern will use NAEP and other federal datasets to explore aspects of student experiences with large-scale assessments. Potential topics may include: an examination of student experiences with the NAEP assessment, an exploration of student actions on the assessment by subgroup (e.g., region, locale, gender), and the development of a data story to display variations in student actions on the assessment.

The ideal applicant will possess an interest in communicating statistical information; experience creating advanced, interactive data visualizations; and demonstrable technical proficiency in their chosen programming framework.

Selected projects from prior interns in this area include:

• Developing interactive data stories that accurately convey statistically valid insights drawn from combining NAEP with other federal and state data sets.
• Developing interactive visualizations allowing the comparison across states and time of student access to high-speed internet and computer resources over the last decade.

Digitally based assessments furnish abundant opportunities for the use of AI tools and methods to advance students’ test-taking experiences. They also hold the potential to support and transform various elements of the assessment cycle, including item generation, item scoring, reporting advancement, and more.

This new topic area is looking for applicants with a solid foundation in machine learning, data analysis, and ethical AI principles to explore the innovative use of artificial intelligence in NAEP assessments within this digital landscape. As part of our dynamic team, you will be at the forefront of harnessing AI’s power to not only enhance learning outcomes but also contribute to educational measurement. You will have the unique chance to work on cutting-edge projects that leverage AI to significantly improve the accuracy, efficiency, and fairness of educational assessments.

A potential topic may involve developing AI algorithms and natural language processing (NLP) models to assist in the creation of high-quality assessment items and content augmentation.