The National Council on Measurement in Education (NCME) and American Educational Research Association (AERA) Annual Meetings are taking place in Philadelphia on April 11–14, 2024. In this post, we’re highlighting a small selection of presentations, trainings, and demonstrations from NAEP researchers in the NCES Center for Process Data and the EdSurvey Team, as well as some NAEP Doctoral Internship alumni who are participating in these Annual Meetings. 

The theme of the 2024 NCME Annual Meeting is “Reconceptualizing Measurement Theory and Practice to Reduce Inequities.” The NAEP research and development community will want to check out the following presentations from members of the NCES Center for Process Data and the EdSurvey Team, whose names are indicated in bold.  

• ShinyBHB: A Shiny App for Bayesian Historical Borrowing in Large-Scale Assessments 
  Innovation Demonstration I: April 12, 10:05 to 11:05 am 
  Weicong Lyu, Shaojie Ye, Jianshen Chen, Sinan Yavuz (Internship Alum), and David Kaplan 

• “ShinyBHB is a comprehensive application package built on the R package Shiny to implement Bayesian historical borrowing for single-level, multilevel, and longitudinal data. It covers Bayesian models with no borrowing, pooling, aggregated data priors, power priors, commensurate priors, and dynamic priors, with flexible model building and various output formats available.” 

• Generative AI Chatbot for Large-Scale Assessment Data Analysis 
  Innovation Demonstration I: April 12, 10:05 to 11:05 am 
  Sinan Yavuz (Internship Alum), Blue Webb, Paul Bailey, Ting Zhang, Yuqi Liao, and Emmanuel Sikali 

• “We fine-tuned the generative AI model and developed specific chatbots to help coding for large-scale assessment (LSA) data analyses. The chatbot focuses on helping with NCES LSA data analysis programming with the R package EdSurvey. We focused on methods that write the codes for analyses and debug the written codes. 

• Process Mining for Mapping Response Sequences: NAEP Grade 4 Math Item 
  Electronic Board Session I: April 12, 1:15 to 2:45 pm 
  Radhika Kapoor (Internship Alum) and Juanita Hicks (Internship Alum) 

• “Process data collected in NAEP 2017, such as clicks and interaction with items, is analyzed for one Grade 4 Math item. This study uses a Fuzzy Miner algorithm to extract and distinguish action sequences of students who got the item correct, partially correct, or incorrect.” 

• Generative AI Chatbot As NAEP Helpdesk 
  Innovation Demonstration II: April 13, 1:15 to 2:45 pm 
  Ting Zhang, Yuqi Liao, Paul Bailey, Blue Webb, Sinan Yavuz (Internship Alum), and Emmanuel Sikali 

• “We fine-tuned the generative AI model and developed a chatbot to help with NAEP data and methodology questions. Our goal is to improve accessibility, efficiency, and user satisfaction in accessing and understanding NAEP data by implementing a generative AI chatbot. We focused on methods that eliminate misinformation from chatbot output.” 

• Reducing Data Usage Barriers: Using Bookdown for Flexible, Open-Source Data Documentation 
  Innovation Demonstration II: April 13, 1:15 to 2:45 pm 
  Karen Yi, Grant Adams, Juanita Hicks (Internship Alum), and Ruhan Circi 

• “Data releases can involve dozens of data files and accompanying documentation, which can quickly become confusing. To remove barriers to data usage, reduce time spent in user support, and promote collaboration among data users, we centralized documentation using bookdown, an R package that builds distributable, living books from R Markdown.” 

• Estimating Student Achievement in Large-Scale Assessments: Can Latent Regression Substitute Plausible Values? 
  Alternative Ways of Estimating Performance in Large Scale Assessments: April 13, 4:55 to 6:25 pm 
  Danielle Siegel (Internship Alum), Ting Zhang, Paul Bailey, and Sinan Yavuz (Internship Alum) 

• “Direct estimation (DE) is a latent regression approach alternative to the use of plausible values to analyze large-scale assessment data. Findings about how well DE performs are mixed. A comparison with simulated data suggests that DE is a suitable alternative to the plausible value approach.” 

• Linking NAEP and TIMSS Scores to Compare US States with TIMSS Countries 
  Large Scale Assessment: April 14, 7:45 to 9:15 am 
  Paul Bailey, Ting Zhang, Huade “Howard” Huo, Charles Blankenship, and Blue Webb 

• “The study linked NAEP and TIMSS scale scores to compare US states with TIMSS countries. Building upon existing linking techniques, we enhanced the techniques by employing a Taylor series approach. This refinement enables us to make comparisons between US states and TIMSS countries, rendering the analyses feasible and replicable.” 

• EdTalk: Intelligent Conversational Agent for Informational Retrieval from Education Reports 
  Innovation Demonstration III: April 14, 9:35 to 11:05 am 
  Bhashithe Abeysinghe (Internship Alum), Abhinav Cheruvu, and Ruhan Circi 

• “EdTalk is a conversational agent that utilizes large language models to conduct document retrieval from education reports by referring to its knowledge base. It generates factual answers by addressing the issue of hallucination. EdTalk aims to make information more accessible to a wide range of users.” 

• Potential Effect of Absenteeism on Declines in Students’ Performance in NAEP 2022 
  Deciphering US Students’ Post-COVID Performance Results from NAEP: April 14, 9:35 to 11:05 am 
  Sinan Yavuz (Internship Alum), and Young Yee Kim 

• “[This] study investigates the role of absenteeism in score declines between 2019 and 2022, using multiple regression analyses.” 

• Deeper Dive Into the Relationship Between Absenteeism and NAEP Performance 
  Deciphering US Students’ Post-COVID Performance Results from NAEP: April 14, 9:35 to 11:05 am 
  Young Yee Kim, Sinan Yavuz (Internship Alum), and Xiaying “James” Zheng (Internship Alum) 

• “[This] study examines the absentee rate change by achievement level and contribution of the change to score declines using Blinder-Oaxaca decomposition.” 

• Applying LASSO Regression to Understand House Effect on PIRLS 2021 Turkey Results 
  Models for International Assessments: April 14, 11:25 am to 12:25 pm 
  Serife Okay, Sinan Yavuz (Internship Alum), and Selhattin Gelbal 

• “This research explores the application of LASSO regression to the PIRLS 2021 Turkey dataset. Given the observed multicollinearity among the selected variables, LASSO, a regularization technique, is employed to derive more precise insights. The analysis also uses the five plausible values (PVs) provided in the dataset to ensure robust conclusions.” 

• Using Process Data to Assess Item Readability Impact on Accessibility Feature Usage 
  Process Data: April 14, 1:15 to 2:45 pm 
  Adam Hearn, Burhan Ogut, and Ruhan Circi 

• “Using process data from the NAEP G8 Math assessment, this study explored the relationship between item readability and accessibility feature usage. Using a random-effects approach, we found that more complex text was associated with varying usage of certain accessibility features, with significant differential effects for students with disabilities.” 

The theme of the 2024 AERA Annual Meeting is “Dismantling Racial Injustice and Constructing Educational Possibilities: A Call to Action.” The NAEP research and development community would benefit from the following professional development course hosted at the AERA Annual Meeting by members of the EdSurvey Team, whose names are indicated in bold. 

• PD24-07: Using R and Generative AI for NAEP Data Analysis 
  Professional Development Course: April 13, 7:45 to 11:45 am 
  Ting Zhang, Paul Bailey, and Sinan Yavuz (Internship Alum) 

• “This 4-hour mini-course will introduce the unique design features of the National Assessment of Educational Progress (NAEP) data to researchers and provide guidance in the required data analysis strategies, including the selection and use of appropriate plausible values, sampling weights, variance estimation procedures (i.e., jackknife approaches) and the generative AI chatbot specifically designed for NAEP data analysis. The course will provide participants with hands-on practice training in analyzing public-use NAEP data files using the R package EdSurvey, which is developed for analyzing national and international large-scale assessment data with complex psychometric and sampling designs, and generative AI chatbot, which is fine-tuned with NCES-approved materials, such as technical documentation on the web, published technical memos and reports.” 

Whether you’re a NAEP Doctoral Internship Alum or just interested in the work our alumni are doing, check out this list showing just how many will be found as participants or presenters in the 2024 Annual Meetings: 

NAEP Doctoral Internship Alumni at 2024 NCME Annual Meeting: 

• Danielle Siegel (2023 cohort)

• Radhika Kapoor (2023 cohort) 

• Yichi Zhang (2022 cohort) 

• Bhashithe “Bash” Abeysinghe (2021 cohort) 

• Xin Qiao (2020 cohort) 

• Nixi Wang (2019 cohort) 

• Sinan Yavuz (2019 cohort) 

• Tong Wu (2019 cohort) 

• Juanita Hicks (2018 cohort) 

• Mingqin Zhang (2018 cohort) 

• Youmi Suk (2018 cohort)  

• Xiaying “James” Zheng (2017 cohort) 

NAEP Doctoral Internship Alumni at 2024 AERA Annual Meeting: 

• Haobai Zhang (2021 cohort) 

• Ayse Cobanoglu (2020 cohort) 

• Huan “Hailey” Kuang (2020 cohort) 

• R. Noah Padgett (2019 cohort) 

• Sinan Yavuz (2019 cohort) 

• Tong Wu (2019 cohort) 

• Youmi Suk (2018 cohort) 

• Maisaa Taleb S. Alahmadi (2017 cohort) 

• Xiaying “James” Zheng (2017 cohort) 

You can still register for the 2024 NCME Annual Meeting and find a program with session details here. You can also find more specific session details and register for the 2024 AERA Annual Meeting here. If you’re interested in joining the ranks of our NAEP Doctoral Internship alumni, you can learn more about the program here; keep following the NAEP R&D Hub to stay up to date and be the first to apply for the next available cohort!