The Every Student Succeeds Act (ESSA) requires all Title I schools to use evidence-based educational activities. It is crucial that administrators understand not only how the law defines evidence-based, but also how to apply that definition to their decision-making process when selecting the resources that will meet their district’s needs.
This web seminar featured a discussion about the practical implications of ESSA for administrators, and how to analyze and apply education research to help make more effective strategic decisions in a district.
Speakers
Luci Willits
Associate Vice President, Policy
Curriculum Associates
Kristopher John
Vice President, Online Assessment
Curriculum Associates
John Lovato, Ed.D.
Assistant Superintendent, Educational Services
Rosemead School District (Calif.)
Luci Willits: ESSA is a new law, passed just a few years ago, and now it’s game time. Part of this implementation is a focus on efficacy. It’s about showing how students are achieving versus all the inputs that get us there. The law actually is very specific about what “evidence-based” is, but it’s up to the district to decide what that looks like.
There are four evidence-based levels that are outlined in the law. The first three are used for school improvement funds and levels 1 through 4 are for Title I funds. The levels are determined by the type of study—is it an experimental or randomized control study or quasi-experimental study?
Kristopher John: The levels imply a higher quality. Level 1, which would be considered the benchmark for strong evidence for an educational intervention, involves well-designed or implemented experimental or randomized control trials.
Quasi-experimental studies—which tend to be based on natural experiments, available data and correlational studies—are considered somewhat weaker in terms of demonstrating that there is causal evidence for an intervention. Nevertheless, a lot of the information you’ll see from providers of interventions and a lot of the research will actually fall into levels 2 and 3.
Luci Willits: It’s important to note that there’s not some great and powerful Oz who’s decided what is evidence-based and what is not. There’s no official approval process for evidence-based solutions under ESSA. The federal government does not decide; there’s no arbiter or federal board. It’s up to districts to ask the right kinds of questions about evidence-based claims. Here are five questions you should be asking when a product claims to be a certain level:
1. When was this study conducted? The big issue with how recently a study was conducted is that the educational landscape is always changing: your state standards, your expectations for students, the curriculum you’re implementing, and even the product you’re evaluating may have changed since the study was conducted.
2. How large was the sample size for the study, and did it include a diverse set of students, including subgroups? Do those demographics match your district’s?
3. Was the study based on current content and standards? You may discover that some of the programs claiming to be level 1 through 4 were based on previous standards.
4. What do the results say? Were they favorable? Get into some of the statistics, because they can be quite confusing—especially the effect size.
5. Does the product truly help kids? And how will it help your kids, in your school district?
John Lovato: One of the things we look for is whether the program will be relevant to the student population. Will it address the areas where our student population struggles? Next we look at the evidence. We want to look at other districts similar to ours—how have their students performed after using these programs? Often I’ll get on the phone and speak directly with people in my position in those districts to ask about successes and challenges.
Another area is resources. It’s important to ask about the total cost for all the different components of a program. Do we have the funding to sustain the program, not only for now but over several years? And can we support ongoing coaching and professional development to fully implement the program?
In our district, we tend to operate in levels 2 and 3, and much of the time in level 4. We rely a lot on the review and the evidence provided by the state of California and the California Department of Education. They do a good job evaluating programs. And we ask the right questions about evidence claims. We look at any research that’s out there on the programs and ask the questions that Luci mentioned.
Kristopher John: Also remember that some studies are done under fairly idealized conditions. The more controlled the study is, the less it might reflect the actual concerns and considerations that go into implementing a program in a real district at scale and over a sustained period of time. You want to look critically not only at the quality of evidence being provided, but also at how well you think your district can replicate the conditions that were used in the research.
Positive strong effect sizes are important, but if they require a depth of support that the district can’t sustain, then you’re probably not going to see ideal results. In some cases, the level 2 and level 3 studies—because they’re messy and they use real-world data—reflect the chaos of the real world in a way that maybe a randomized control trial study can’t replicate.
To watch this web seminar in its entirety, please visit districtadministration.com/ws050318
