1. What are evidence-based strategies?
Evidence-based strategies are programs, practices or activities that have been evaluated and proven to improve student outcomes. Districts can have confidence that the strategies are likely to produce positive results when implemented.
The term “evidence based” is not new. It has been used in the field of medicine since 1996 and is defined among medical professionals as “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of the individual patient.”[1] When thinking about the term from an education perspective, “patient” could be substituted with “student.”
“Evidence based” has been catapulted into the education arena by the Every Student Succeeds Act (ESSA). Federally, this shift emphasizes the importance of making decisions that are based upon a rigorous evaluation. Prior to ESSA, the Elementary and Secondary Education Act (ESEA) consistently used “research based” when describing strategies. No Child Left Behind (NCLB) used “scientifically-based research” as its threshold. “Evidence based” represents a higher expectation.
Note that resources created prior to enactment of ESSA (before July 2016) might have references to being “evidence based,” but that does not necessarily mean they meet ESSA’s definition of “evidence based.”
Back to Top
2. Why do evidence-based strategies matter?
An educator’s top priority is success for each and every student. Fulfilling this priority means that selected strategies must yield maximum return. This is especially important as educators support students with diverse needs, and administrators are faced with limited resources.
But using evidence to inform the selection of a strategy is not the only thing that matters. To achieve greatest impact on student outcomes, districts should carefully consider strategies that are:
- Aligned with the district’s or school’s specific needs and educational context. In other words, district or school leaders have given careful thought to deploying the strategy in a way that recognizes its unique local characteristics;
- Part of a cohesive improvement plan. This means the strategy is integrated into the district’s or school’s systemic improvement plan and complements its efforts;
- Implemented with fidelity. This means that the district or school is committed to the long-term follow through of the strategy and is careful to understand its intent and preserve the integrity of its design.
Evidence of a successful strategy is determined through rigorous research and evaluation. If such evidence does not yet exist, districts should be prepared to evaluate the effectiveness of their selected strategies.
Using evidence to determine the most effective strategy — coupled with a systemic improvement plan and sustained implementation — goes a long way to enable success for each and every student.
Back to Top
3. How do evidence-based strategies fit into a school’s systemic plan for continuous improvement?
Selecting an evidence-based strategy is one important part of an effective cycle of continuous improvement. The cycle also should include:
- An initial needs assessment to help ensure the strategies are sensitive to the district’s or school’s specific needs;
- An alignment test to ensure the strategy is working in service of the district’s or school’s systemic continuous improvement plan;
- Local data analysis and evaluation to determine if the strategy is working as intended.
Back to Top
4. How are evidence-based strategies differentiated?
ESSA (Section 8002) and the U.S. Department of Education’s Non-Regulatory Guidance: Using Evidence to Strengthen Education Investments outline four levels of evidence. Level 1 represents the strongest level of evidence and, therefore, the strongest level of confidence that a strategy will work. The table below includes ESSA’s definition for each of the four levels, along with a practical interpretation of each level.
Level
|
ESSA definition
|
What does it mean?
|
| Level 1 |
Strong evidence from at least one well-designed and well-implemented experimental study. |
Experimental studies have demonstrated that the strategy improves a relevant student outcome (reading scores; attendance rates). Experimental studies (Random Control Trials) are those in which students are randomly assigned to treatment or control groups, allowing researchers to speak with confidence about the likelihood that a strategy causes an outcome.
Well-designed and well-implemented experimental studies meet the What Works Clearinghouse (WWC) evidence standards without reservations.
The research studies use large, multi-site samples.
No other experimental or quasi-experimental research shows that the strategy negatively affects the outcome.
Researchers have found that the strategy improves outcomes for the specific student subgroups that the district or school intends to support with the strategy.
|
| Level 2 |
Moderate evidence from at least one well-designed and well-implemented quasi-experimental study. |
Quasi-experimental studies have found that the strategy improves a relevant student outcome (reading scores, attendance rates). Quasi-experimental studies (Regression Discontinuity Design) are those in which students have not been randomly assigned to treatment or control groups, but researchers are using statistical matching methods that allow them to speak with confidence about the likelihood that a strategy causes an outcome.
Well-designed and well-implemented quasi-experimental studies meet the What Works Clearinghouse (WWC) evidence standards with reservations.
The research studies use large, multi-site samples.
No other experimental or quasi-experimental research shows that the strategy negatively affects the outcome.
Researchers have found that the strategy improves outcomes for the specific student subgroups that the district or school intends to support with the strategy.
|
| Level 3 |
Promising evidence from at least one well-designed and well-implemented correlational study. |
Correlational studies (studies that can show a relationship between the strategy and outcome but cannot show causation) have found that the strategy likely improves a relevant student outcome (reading scores, attendance rates).
The studies do not have to be based on large, multi-site samples.
No other experimental or quasi-experimental research shows that the strategy negatively affects the outcome.
A strategy that would otherwise be considered Level 1 or Level 2 except that it does not meet the sample size requirements is considered Level 3.
|
| Level 4 |
Demonstrates a rationale based on high-quality research findings or positive evaluation that such activity, strategy or intervention is likely to improve student outcomes or other relevant outcomes. |
Based on existing research, the strategy cannot yet be defined as a Level 1, Level 2 or Level 3.
However, there is good reason to believe — based on existing research and data — that the strategy could improve a relevant student outcome.
Before using a Level 4 strategy, districts should:
- Explore Existing Research: Why do we believe this strategy will meet our needs?
- Develop a Logic Model: How will the strategy improve student outcomes?
- Plan to Evaluate: How will we know that the strategy is improving student outcomes?
|
Back to Top
5. How do local characteristics and needs factor into the levels of evidence?
While a strategy may have been proven to work for the general student population, we cannot assume that the same strategy will have the same effect on specific student subgroups.
A strategy can only be considered a Level 1 or Level 2 strategy for a district or school if the research shows that the strategy improves student outcomes for the student subgroup that the district or school intends to support. If, for example, a district or school has identified a need to offer additional supports to their students with disabilities, a Level 1 or Level 2 strategy for that district will be one that has been proven to work for students with disabilities.
Considering the unique needs of specific student subgroups is a valuable consideration regardless the level of evidence associated with a strategy. There may be cases where Ohio will require districts to take those unique needs into consideration when using Level 3 or Level 4 options for school improvement or grant opportunities. These cases will be identified and detailed on a case-by-case basis.
Back to Top
6. Are there other important considerations to keep in mind while selecting evidence-based strategies?
Beyond the technical definitions of levels, there are other important considerations to keep in mind while selecting evidence-based strategies, including:
- How much will the strategy cost to implement? Cost of implementation is not directly factored into the technical definitions of evidence-based strategies. However, start-up and sustainability cost is certainly a factor that districts can and should consider when deciding which evidence-based strategies are best suited to meet their districts’ needs.
- Can the strategy be implemented with fidelity? Evidence-based strategies may be less effective if they are not carried out as intended.
Back to Top
7. What is the role of level 4?
Level 4 enables districts to innovate and explore new strategies that have strong potential for improving student outcomes. Often, the most promising innovations in education bubble up from the local level.
While there will be circumstances where districts will be required to use strategies identified with strong (Level 1), moderate (Level 2) or promising (Level 3) evidence, there also will be opportunities for districts to leverage Level 4 strategies. Options for using Level 4 strategies to address school improvement requirements or grant opportunities will be identified and detailed on a case-by-case basis.
Before using a Level 4 strategy, districts should:
- Explore Existing Research: Why do we believe this strategy will meet our needs?
- Develop a Logic Model: How will the strategy improve student outcomes?
- Plan to Evaluate: How will we know the strategy is improving student outcomes?
Back to Top
8. What is the difference between “evidence based” and “research based”?
The terms “evidence based” and “research based” are frequently used interchangeably, but they are different — and it is important to understand the difference.
A strategy that is evidence based likely also is research based; however, the reverse is not always true. A program or strategy — especially if it is newly developed — may be research based but not meet the formal definitions of evidence based.
For a strategy to be considered “evidence based,” its efficacy must have been evaluated by someone other than just the people or organizations that developed the strategy. The outcome of the evaluation(s) will determine what, if any, level of evidence the strategy meets.
While generally there is research that goes into the development of a strategy, it must be evaluated for efficacy, as outlined by ESSA, to fulfill Ohio’s state or federal requirements related to evidence-based strategies.
Back to Top
9. What is Ohio doing to support districts’ use of evidence-based strategies?
The Department plans to take the following steps:
Post resources and tools aimed at operationalizing the “evidence-based” approach, including:
- Guidance on how federal law and state policy contribute to the definitions of the evidence-based levels that Ohio will use;
- Step-by-step guidance on how to leverage existing resources, such as the What Works Clearinghouse or other existing clearinghouses and reports, to identify evidence-based strategies.
The Department’s website will be updated on a regular basis to reflect current information on Ohio’s approach to using evidence in education.
Launch Ohio’s Evidence-Based Clearinghouse
As part of its ESSA plan, Ohio committed to developing an evidence-based clearinghouse, a system that will support districts with the knowledge, information, tools and resources they need to identify, select and implement evidence-based strategies for improving student outcomes. This resource will be available to districts in the summer of 2018.
Ohio’s evidence-based clearinghouse will be designed with several key guiding principles in mind. The clearinghouse will be:
- Practitioner-Focused. Ohio school administrators and teachers are the primary audience for the clearinghouse. While academic research plays an important role in establishing levels of evidence, the information, resources and connections to other systems that are incorporated into the clearinghouse must be clear and actionable at the local level.
- Integrated with Ohio’s School Improvement System. The evidence-based clearinghouse will serve as a critical resource for districts as they complete needs assessments, identify areas for school improvement and identify relevant strategies supported by appropriate levels of evidence. To the degree possible, the evidence-based clearinghouse should be seamlessly integrated into this process.
- A Dynamic, Growing Resource. The evidence-based clearinghouse is not meant to be a stagnant archive of information but rather a dynamic, living tool that provides districts and educators with resources they can use on a regular basis as they work to improve student outcomes.
Promote Peer-to-Peer Improvement Network
As part of its broader efforts to develop an evidence-based framework for school improvement, Ohio will foster Peer-to-Peer Improvement Networks to encourage partnerships and opportunities for educators to collaborate across district boundaries, and across subjects that support a well-rounded education, to fashion solutions to common challenges.
The evidence-based clearinghouse will be directly connected to the Peer-to-Peer Network Improvement efforts as it will allow districts to learn more about the strategies and strategies that other districts and schools are leveraging to address specific needs.
Back to Top
10. Are there additional resources educators can use to learn more about using evidence in education?
Back to Top