The NAEP R&D program recently reconnected with former intern Yichi Zhang at the National Council on Measurement in Education (NCME) annual meeting, where AIR was presenting a study using the ROME (Bayesian Region of Measurement Equivalence) method she developed during her internship. We followed up after the conference to get her perspective on the NAEP Doctoral Internship program, which is gearing up for its next cohort in June. Read on to learn about Yichi’s experience, including advice for prospective interns and more about her ROME method for advancing equity in educational assessment.

Yichi, a 5th-year Ph.D. student in the University of Southern California’s Quantitative Methods and Computational Psychology program, joined the 2022 cohort of NAEP Doctoral Internship Program interns from Los Angeles. She took the time to answer a few questions about her experience as a NAEP doctoral intern.

1. Can you share some highlights from your experience as an intern with us?

“One of the highlights of my internship was to meet researchers and fellow interns from the NAEP team in person at conferences such as NCME and IMPS. Since my internship was remote in 2022 due to the pandemic, these face-to-face interactions were particularly valuable. I immensely enjoyed learning about the recent research and current work of NAEP researchers and fellow interns, as well as connecting with them in between conference sessions.”

2. What specific skills or knowledge did you gain during your internship that you find most valuable?

“One of the most valuable skills I gained during my internship was the ability to communicate research findings effectively to individuals with diverse backgrounds and training. It’s easy to fall into the trap of assuming shared knowledge when discussing research. However, I’ve learned that a successful researcher not only conducts quality research but also possesses strong communication skills to convey findings clearly and comprehensively to a broader audience.”

3. Can you share any advice or tips for current or prospective interns?

“One tip that I have is to maintain an open mind and embrace every opportunity to learn. Researchers from the NAEP team are very friendly and willing to share their experiences, so I highly recommend arranging one-on-one coffee chats with them to gain insights into their daily work and hear their career advice.

“Additionally, I encourage interns to maintain connections with the team even after the internship ends. While the internship may last only 10 weeks, the connections you build can be long-lasting and beneficial throughout your career.”

4. When do you anticipate graduating, and how has your internship experience contributed to or influenced your professional development and academic pursuits?

“I plan to graduate this summer. My internship experience has laid a solid foundation for me to continue pursuing fairness-related research on large-scale educational assessment. I also really appreciate the connections I built with NAEP researchers and fellow interns. These relationships have not only enriched my understanding of the field but also provided me with a supportive network that I can turn to for guidance and collaboration in the future.”

5. Could you provide an overview of your dissertation topic? How has the internship shaped or enhanced your work on your dissertation?

“My dissertation is about evaluating the practical significance of measurement bias in large-scale educational assessment. During my internship, I implemented the methodology that I had previously proposed on the NAEP dataset and provided validity support for NAEP noncognitive items. Inspired by this internship experience, I developed some promising extensions of my methodology to improve its efficiency with large-scale educational assessments, which became a part of my dissertation.”

6. One of AIR’s presentations at this year’s NCME was based on the ROME (Bayesian Region of Measurement Equivalence) method you developed; it received high praises for its methodological soundness and novelty from the discussant, the esteemed Dr. Mark Reckase. Could you briefly describe the development, application, and significance of this method conceptually for a general audience?

“I developed the ROME method to address an important challenge in educational assessments: ensuring fairness across diverse student subgroups. In educational assessments, one important validity assumption is that the assessment measures student ability equivalently across subgroups. However, as every individual is different, people from diverse backgrounds might react differently to test items due to their unique experiences, leading to nonequivalence in measurement and potential bias. The ROME method aims to quantify the practical impact of such bias on interpretable metrics, bridging a gap in the current statistical framework that often overlooks the degree and impact of bias. This method will make it easier to translate findings about bias into actionable insights for future assessment use.

“Conceptually, the ROME method offers several advantages over traditional approaches. Firstly, it focuses on whether the nonequivalence in measurement is within a tolerable threshold, which enables test evaluators to support the validity of the test. Secondly, it provides a visual representation of the practical impact of bias across different levels of latent ability, offering valuable insights that can inform policy decisions related to test usage. In short, the ROME method serves as a complementary tool to traditional methods, offering a nuanced approach to quantifying and addressing measurement bias, thereby promoting fairness and equity in educational assessments.”

7. As you approach graduation, what are your career goals, and how has your internship with us shaped these ambitions?

“I hope to achieve fair and accurate educational assessment for all students, which motivates me to conduct fairness-related research.

“My internship experience has been invaluable to me in several ways. First, it provided me with the opportunity to apply my proposed methodology to real-world data. This hands-on experience allowed me to critically assess the strengths and limitations of my method in operational settings. It also deepened my understanding of this method’s practical implications and impact on downstream analysis. Second, the guidance and support from my mentors at NAEP have been extremely helpful. I learned how to interpret the research findings from a practical perspective and how to communicate the results effectively with stakeholders. In summary, this internship set a solid foundation for me to continue my work in fairness-related research.”

We congratulate Yichi on her successful internship experience and research contributions and look forward to staying connected with her as she progresses in her career. If you find Yichi’s account of her experience and growth compelling, consider checking out the NAEP Doctoral Internship Program for yourself. You could be part of the next cohort of graduate students working directly with NCES researchers and participating in methodological developments and secondary analyses using NAEP data.

While internship applications aren’t currently open, applications for our upcoming NAEP Data Training Workshop are! Follow the link to learn more about the workshop and apply by May 12. To stay up to date on NAEP research opportunities and be the first to apply for our next internship cohort, keep checking in on the NAEP R&D Hub and sign up for our mailing list.