NAEP RESEARCH AND DEVELOPMENT

The NAEP statistical and psychometric methods research team applies advanced statistical and psychometric methods to investigate a wide range of methodological and substantive issues related to measurement in large-scale assessments. Potential topics for the internship program include missing data, analysis of complex sample data, issues in multistage testing, multiple-group analysis, and test-taking behaviors (e.g., speededness and rapid guessing). Applicants with knowledge and research experience in structural equation modeling (e.g., factor analysis, path analysis, and finite mixture models), item response theory (e.g., calibration, scoring, linking, plausible value generation, and multidimensional/multigroup IRT), and response time modeling are strongly encouraged to apply. Proficiency in flexMIRT, Mplus and/or various SEM/IRT R packages is expected. Familiarity with programing language such as Python, Bayesian modeling with Stan, and/or machine learning algorithms is a plus.

Selected psychometrics & statistical methods projects from prior interns include:

• Applying response time and response data joint modeling to detect speededness
• Exploring gender-related differential item time functioning in digitally based assessments
• Examining the effect of digital familiarity on writing performance using multiple-group analysis
• Designing a novel MST assembly method for a test with many subscales

Process data have the potential to provide valuable insights into students’ responding processes and behaviors as well as aid in evaluating how students interact with the assessment items, system functions, and exam delivery platform, and offer a wide range of projects. Some areas of research for process data are:

1. General process data research: Potential projects include data privacy and quality projects (e.g., developing a systematic approach to create process datasets and ensure information quality); implementing methods for the extraction of useful information from process data (e.g., pattern recognition, uncertainty modeling); and exploration of new methods for analyzing process data. Strong statistical skills and the ability to work effectively with large and complex data as well as good programming skills (R and/or Python) are required. Psychometrics skills and experience in advanced statistical modeling/machine learning methods are preferred.
2. Assessment item and delivery platform development process data research: Potential projects include investigating student interactions with digital tools such as calculators and equation editors, investigating if interactive item components (IICs) are being used as intended, determining whether students at different achievement levels interact and use digital tools differently, and investigating the validity of scores using students’ interactions with items and/or digital assessment platforms. The intern should have an interest in large scale K-12 mathematics assessments and in learning how process data can be used to inform item and platform development along with other operational decisions. The intern is expected to have good R programming skills and experience with text-mining, image processing, or image recognition is preferred.

Selected process data projects from prior interns include:

• Understanding Students’ Problem-Solving Processes via Action Sequence Analyses
• Item profiling: Insights into characteristics and design of each item
• A Social Network Analysis of Answer Change Behavior Using NAEP Process Data
• Insights from students’ computations using the calculator
• A comparison of student disengagement across years
• Whether items can be reconstructed from the process data

The Policy-Relevant research team uses NAEP data to explore research questions informed by education, economics, psychology, sociology, and any other related disciplines that have the potential to affect policy and practice in K-12 education. Potential topics for the internship program include but are not limited to analyzing trends in noncognitive outcomes and examining the relationship between noncognitive traits, NAEP performance, and college outcomes; understanding the mechanism behind students who are academically resilient and identifying factors to help these students continue to be successful; analyzing trends in achievement gap among students from different socio-economic status (SES) backgrounds and comparing results across multiple assessments (e.g., the Trends in International Mathematics and Science Study); and examining relationship between NAEP achievement and postsecondary outcomes to identify college preparedness indicators. Applicants with an interest in these and other topics in the policy-relevant research area and experience with multi-level analysis, structural equation modeling, factor analysis, survival analysis, propensity scores, econometric techniques, and other advanced statistical modeling methods are encouraged to apply. Familiarity with Stata, Mplus, and/or R is preferred.

Selected projects in policy-relevant research projects from prior interns include:

• The role of mathematics motivation, self-other identity discrepancies and STEM course-taking in grade 12 mathematics achievement using the overlap sample of NAEP and the High School Longitudinal Study (HSLS:09)
• The relationships between student experiences and NAEP technology and engineering literacy achievement
• College preparedness research using NAEP data or informing NAEP
• Examining the relationship between mathematics growth and growth in working memory in the early grades using the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011)

A project in this topic area will focus on expanding the nature and scope of reports based on NAEP data by combining communication and information science skills for highly-engaged constituencies who lack expertise in data analysis. Such constituencies may include policymakers, parents, and teachers. The intern will be part of a multidisciplinary team, composed of NCES and AIR staff, involving analysts, graphic designers, measurement scientists, and programmers.

Additionally, the intern will explore rich sets of data collected by surveys at the National Center for Education Statistics that can be combined and synthesized with data from other federal surveys, such as the U. S. Census, or other studies, such as K-12 students’ learning experiences during the SARS-CoV-2 pandemic. The ideal applicant will possess an interest in communicating statistical information; experience creating advanced, interactive data visualizations; and demonstrable technical proficiency in their chosen programming framework.

Potential projects and selected projects from prior interns in this area include:

• Developing interactive data visualizations that accurately convey statistically valid insights drawn from combining NAEP and other federal data sets.
• Developing data visualizations that transform process data from the 4th-grade NAEP reading assessment to a diagnostic display to support reading instruction.
• Documenting best practices and innovative uses of NAEP reports at the state, district levels with a focus on cross-team/varied-expertise stakeholder/cross-discipline use of NAEP reports for local decision-making.