R&D Hub

This month, we are excited to share the latest research from the NAEP R&D community, focusing on the evaluation of large language models (LLMs). “The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approaches” will be presented at the LLM4Eval workshop at the 2024 Association for Computer Machinery Special Interest Group on Information Retrieval (ACM SIGIR) conference on July 18, 2024. This paper from AIR researchers Bhashithe Abeysinghe and Ruhan Circi explores innovative approaches to evaluating custom AI applications, addressing a crucial barrier to faster progress in generative AI.

NAEP researcher David Bamat recently published a new research article, “Closing Reporting Gaps: A Comparison of Methods for Estimating Unreported Subgroup Achievement on NAEP,” in Measurement: Interdisciplinary Research and Perspectives.

Explore the latest insights in research and development from NAEP researchers in the R&D program. Given the current buzz around AI, this month we’re excited to share a working paper on the evaluation of chatbots by Ruhan Circi and Bhashithe Abeysinghe, NAEP researchers from the American Institutes for Research (AIR). The full text of the working paper is included below. Don’t miss out on these valuable perspectives – subscribe now to stay ahead in technology and innovation!

Those interested in process data in digitally based assessments (DBAs) will want to check out the newest addition to the Focus on NAEP series: “Exploring Process Data in TEL.” This examination of process data from the 2014 Technology and Engineering Literacy (TEL) assessment explores how data collected from digitally based assessments can be used to investigate digital familiarity and efficiency.

Paul Bailey and Blue Webb, members of the EdSurvey team, recently published an article, “Expanding NAEP and TIMSS Analysis to Include Additional Variables or a New Scoring Model Using the R Package Dire.” Featured in the journal Psych in its special issue on computational psychometrics, this article showcases the new Dire software and its utility for ushering in a new era of data use at the National Center for Education Statistics (NCES).