Assistant Professor Susu Zhang has received funding under an Institute of Education Sciences (IES) research grant awarded to Digital Promise and UIUC to conduct a project entitled, "Analysis of NAEP Mathematics Process, Outcome, and Survey Data to Understand Test-Taking Behavior and Mathematics Performance of Learners with Disabilities". The National Assessment of Educational Progress (NAEP) is a large-scale national assessment, administered every two years, to monitor what US 4th- and 8th-grade students know and can do in various subjects. This project will analyze the computer log data generated from student interactions with the digital NAEP assessment, known as process data, to examine how test-taking behavior relates to math performance for learners with disabilities.
Could you provide some background on the motivation behind this project? What inspired you to take it on?
For decades, assessments such as the NAEP indicated a math achievement gap between students with disabilities and their typically developing peers. With NAEP's transition to digital assessment, there has been growing interest in the potential utility of test-taking process data, coupled with student background and proficiency information, to provide additional insights into how problem-solving behavior (e.g., test-taking strategies, misconceptions, use of accommodations and tools) explains performance differences across demographic groups. Since 2020, the IES, which is the statistics, research, and evaluation arm of the Department of Education, launched several efforts to leverage process data to understand math test-taking behaviors for students with disabilities, including the release of select NAEP process data files for research use and calls for research proposals analyzing this data, to gather evidence that ultimately contributes to the improvement of learning of these students from special populations. Our project was one of these funded exploration grants.
Were there any collaborative efforts or partnerships that played a crucial role in the success of this project/opportunity? Could you tell us about them?
One of the key contributors to this effort is the PI on this proposal, Dr. Xin Wei from Digital Promise. As a quantitative psychology researcher, I am primarily interested in methodological developments, i.e., how should unstructured process data be analyzed to answer educational questions? Xin's experience with both special education and quantitative methods was essential, both for identifying practically relevant research questions that address educators' and policymakers' needs and for communicating our findings to these key stakeholders.
How does this opportunity support the Department of Psychology's and the University's broader goals, values, and strategic initiatives?
This project allowed me to join the university's and the Psychology Department's efforts to promote diversity, equity, and inclusion. There are two types of questions that the current research on NAEP process data helps address: The first is testing equity/fairness. For instance, are students with disabilities navigating the digital testing interface and using various embedded tools the same way as their peers? And are the accommodations they receive (e.g., extended time) effective in providing them with adequate opportunities to demonstrate their knowledge and skills? The second is educational equity, i.e., providing each student with what they need to learn. For instance, can we find patterns from the process data that suggest common challenges/misconceptions among students with disabilities? Identifying these patterns may help educators design evidence-based instructions to close achievement gaps.
Are there people you'd like to acknowledge for their contributions to this opportunity's success?
I would like to acknowledge our Quantitative Psychology student Sunbeom Kwon, who contributed to both the methodology development and the analysis of the NAEP process data. I am also thankful to our Psychology staff members, especially Lori, Sarah, and Firmino, for their support in preparing the federal grant proposal, making sure the project is performed as planned, and applying for the NAEP data license. I am also indebted to many mentors and colleagues who inspired me in research and generously shared their advice on writing grant proposals.
Related People
