As the nation increasingly embraces the science of reading—a term referring to an interdisciplinary body of research about reading and writing—some critics have argued that using assessment results for item-level analysis to support teaching and learning in standards-based English language arts is a “fool’s errand” (Catts, 2022; Shanahan, 2014; Shanahan, 2024).
This skepticism often arises from large-scale testing theory, which traditionally views item analysis as a tool for test developers to evaluate the quality of individual questions based on student responses (Salkind & Frey, 2023). In this context, item analysis is primarily used to refine questions for future tests or to identify and remove ambiguous items in a single test administration.
However, modern item analysis, informed by classroom assessment research—a field rooted in formative assessment, psychometric design, curriculum and instruction, and school leadership—suggests otherwise. Contemporary research indicates that well-designed items and their subsequent analysis can significantly enhance the teaching and learning cycle (Boudett, City, & Murnane, 2013; Brookhart & Nitko, 2019; Gareis & Grant, 2015; McTighe & Ferrara, 2024; Mertler, 2007; Popham, 2020; Shermis & Di Vesta, 2011).
This raises a critical question for educators: Should they follow the perspectives of the science of reading or embrace insights from classroom assessment research regarding item analysis? In this two-part series, we argue that educators need not choose one over the other but should instead adopt an integrated approach.
As Jim Collins (1994, p. 96) articulates, rather than being constrained by the “Tyranny of the OR,” educators can embrace the “Genius of the AND.” This concept involves leveraging the strengths of both perspectives—combining A AND B rather than choosing between them. We believe that educators can and should incorporate the practices of the science of reading while also utilizing insights from classroom assessment research to enhance the teaching-learning process.
We posit that there are responsible and effective ways to apply modern item analysis in ELA. Rather than dismissing it outright, especially given that the Reading Rope model includes standards directly relevant to the concerns surrounding item analysis, educators can benefit from its diagnostic potential (see Figure 1).
Figure 1: Alignment of CCSS ELA Standards to Scarborough’s Reading Rope
Throughout this series, we will examine real-world scenarios where educators engage with item analyses from standards-based ELA assessments in a diagnostic manner. These scenarios align with a research-based diagnostic approach that we will outline in the following sections.
Framework for item analysis
There are five research-based approaches (see Figure 2) that educators can use to leverage item analysis data diagnostically to enhance instruction (Brookhart & Nitko, 2019).
- Identifying Prerequisite Deficits: This approach involves pinpointing foundational concepts that students need to understand before they can benefit from further instruction. For example, if a student struggles to comprehend a passage, it may be due to difficulties with decoding skills.
- Identifying Students’ Errors: This method focuses on identifying specific areas where students struggle, allowing educators to target instruction effectively. For instance, is the student having trouble with vowel teams or CVCe (consonant-vowel-consonant-silent e) word patterns?
- Profiling Student Strengths and Weaknesses: This approach compares student performance within a broad learning outcome area and specific standards relative to their peers. For example, educators might assess whether all students in a particular second-grade class have mastered digraphs compared to other second-grade classes.
- Identifying Unmastered Standards or Objectives: Here, educators use multiple data points to identify gaps in student learning. For instance, if a concept was taught but students still show difficulties, it may indicate a need for reteaching or review.
- Identifying Student Knowledge Structures: This approach examines how students mentally organize concepts, their interrelationships, and the extent to which they can apply content knowledge to solve problems (e.g., using Webb’s Depth of Knowledge, 2006). Did instruction focus on enabling students to transfer knowledge effectively, or was it limited to recall-level understanding?
With this framework in mind, we will apply these approaches to our first scenario in this article and continue exploring three additional scenarios in the second part of this series.
Figure 2
Scenario 1: Identifying Prerequisite Deficits
Mr. Diggery D. Per Dives Below the Surface
Mr. Diggery D. Per, a fourth-grade teacher, is reviewing his spring state ELA assessment data. He feels optimistic that more fourth graders passed the assessment (32%) compared to when they were in third grade (17%). However, the low overall proficiency rates are unacceptable. During item analysis, he notices a recurring trend: students missed text comprehension questions. He decides to explore different comprehension strategies. Surely, these new strategies will improve students’ reading scores. Not so fast.
Although Mr. Per is analyzing data and utilizing evidence-based strategies (Sharratt & Fullan, 2022), the data is masking a larger issue unrelated to the comprehension interventions he is providing. Gough and Tunmer’s Simple View of Reading (1986) provides a theoretical framework where Reading Comprehension is the product of Word Recognition and Language Comprehension. If students struggle in one or both of these areas, their ability to comprehend texts will be compromised (Scarborough, 2001). Could his students have phonics gaps hindering their comprehension?
After analyzing data from a universal screener and conducting further diagnostic assessments, the answer is clear: “Yes!” This scenario underscores the importance of the science of reading and its focus on interconnected reading skills. Mr. Per decides to “dive below the surface” by examining universal screener and diagnostic data, revealing the true source of the comprehension issues.
By applying the prerequisite deficits approach to item analysis, Mr. Per identifies students struggling with comprehension may have phonics skills gaps. Thus, his instruction should target strengthening phonics skills to address these deficiencies. Gaps in foundational skills like phonics often manifest as poor text comprehension; if left unaddressed, they will continue to impede progress, regardless of how much focus is placed on direct comprehension strategies. As Archer & Huges articulate, there is no comprehension strategy powerful enough to compensate for the fact you can’t read the words.” (Archer & Hughes, 2011).
This first scenario is just a quick peek at how to embrace the science of reading and modern item analysis. In our second article, we will continue to explore how to use this information in pursuit of improved reading achievement.
