As the NAEP R&D program prepares for the next cohort of summer interns, we caught up with recent NAEP Doctoral Internship Program alumnus Eric Ho to hear his perspective on the program and the work he’s done with NAEP data.
Eric Ho joined the 2022 cohort of NAEP Doctoral Internship Program interns from the University of California, Los Angeles School of Education’s Social Research Methodology graduate division. He chose to focus on the Data Visualization topic area and was able to present his project for the internship, “Operation REQUISITE – Re-envisioning Educational Quantitative User Information: Shiny Interface to Explore,” at the 2023 National Council on Measurement in Education (NCME) Annual Meeting. Here’s how Eric describes his project:
“In my project, I hoped to make NAEP process data useful not only to researchers but also to practitioners. I thought that this could be done through a framework that provides guidance on how to choose the most useful visualizations to make sense of the process data, based on the questions they wanted to answer. To that end, I built a Shiny application that uses this framework to guide users, allowing them to create unique visualizations of the data that are best suited for their research questions and purposes. We demonstrated the framework and application in investigating and reporting the differences in test-taking behaviors between students performing in the top quartile of the below NAEP Basic achievement level and in the bottom quartile of the NAEP Basic achievement level.”
Eric was very positive about his experience with the internship:
“I had a great time working with the AIR researchers and networking with other interns. I felt that I was able to apply what I had learned in graduate school to effect meaningful change. I also learned a lot about the NAEP and enhanced my technical skills in cleaning and analyzing process data. Because of what I learned, I felt more confident in developing the framework and the Shiny application and even in finishing my dissertation.”
He shared some of the challenges in working with NAEP data that the internship prepared him for:
“I had not been used to working with such fine-grained data. There are a lot of considerations, such as checking for errors in the events and properly aggregating to the desired level. The internship provided me with the technical know-how to analyze the data and an understanding of its context, allowing me to raise and properly answer the right questions.”
Eric also talked about some of the potential areas for future research around his work:
“The framework can be further fleshed out, and the Shiny application can undergo additional testing to ensure an optimal user experience. I think a usability study would allow us to understand the most pressing questions that practitioners would want to answer. That, in turn, would allow us to design our framework and application around them.”
He shared how he feels his work during the internship could help inform policy and practice:
“I think that this tool would increase the accessibility of the NAEP data to the general public, allowing policymakers to better understand the performance of students on the NAEP. Although there have always been presentations on NAEP results, I think that the finer-grained information provided by our work would be novel and provide more actionable insights. For example, teachers can not only see which students are struggling but also in what ways. Are they spending too much time on certain items? Is their elimination strategy faulty? How are the students interacting (if at all) with the provided tools? In turn, our work can hopefully inform NAEP developers in elucidating how students interface with the platform.”
Ultimately, Eric was happy to recommend the program to any interested potential applicants:
“Yes, I would recommend the program. I learned a lot and felt that this program has contributed to my personal and professional growth. I am sure that future interns will enjoy similar benefits.”
We congratulate Eric on his research contributions and look forward to staying in touch with him as he progresses in his career. Stay tuned to the NAEP R&D Hub as we continue to put a spotlight on our interns, alumni, and other vital members of the NAEP R&D community. If you find Eric’s experience and recommendation compelling, consider checking out the NAEP Doctoral Internship Program for yourself. You could be part of the next cohort of graduate students working directly with NCES researchers and engaging in methodological developments and secondary analysis using NAEP data. To be the first to apply to our next internship opportunity, keep checking in on the NAEP R&D Hub and sign up for our mailing list.