This case study looks at two methodologies for turning innovation into institution-wide impact – implementation science and improvement science – through the experience of Baltimore City Public Schools (“City Schools”). Over the last five years, City Schools has implemented two new secondary school-level curricula, Wit & Wisdom for English Language Arts and Eureka Math for mathematics, and introduced improvement science in English Language Arts. The district is among the nation’s 50 largest, with one of the highest shares of students who are Black, Latino, or experiencing poverty. Although it is still early, the district is seeing encouraging improvements and “larger jumps in test scores than many school systems in the state” (Bowie 2020).
City Schools’ experience illustrates how implementation and improvement sciences complement each other and can be woven together for greater impact.
Summary: Lessons for effective implementation and improvement
To learn how City Schools’ experience can inform the use of implementation and improvement sciences, we interviewed over 20 staff and partners and reviewed relevant research. Four categories of lessons emerged:
1. Cultural emphases: Implementation benefits from a demand for urgent, large change; improvement, a commitment to ongoing, evolutionary change.
Implementation: A shared sense of urgency makes large-scale changes seem worthwhile. Often, external factors prompt urgency, like the need to align with new state academic standards. Yet leaders cannot rely on the external context —they must also cultivate a sense of productive disequilibrium, such as by highlighting new materials’ role in the vision for all students’ success.
Improvement: A sustained commitment to iterative learning and refinement can turn evolutionary changes into district-wide progress. Teacher-leaders learn and then engage their peers in improvement science tools, such as empathy interviews and inquiry cycles. Patience Hein, an Assistant Principal, has found that this improvement science work “helps us identify small changes in instruction that might generate large results.”
2. Team structures: Cross-district coherence is vital whether defining roles for implementation or integrating improvement into existing roles.
Both implementation and improvement science emphasize peer networks and teacher perceptions. City Schools leveraged peer leaders to build buy-in for implementing Wit & Wisdom: For instance, teacher-fellows helped build support for the curriculum by “tweeting about what was happening in their classroom,” recalls teaching and learning director Janise Lane.
Similarly, student and community engagement have built the foundation of trust that enables implementation and improvement success. In Baltimore, the need for cultural relevance surfaced early and often. As the district implemented Wit & Wisdom, it worked with the publisher to host events on culturally relevant texts and instruction. The district also launched a teacher-led social studies program that aligns standards with the stories of students and their city. These efforts have “absolutely been beneficial” to success with the materials, says district CEO Sonja Santelises.
Implementation often requires changes to teams and roles to facilitate coherence across schools. For Baltimore, greater coordination between the academic and schools offices has been crucial. So, too, was clearly defining the roles of principals and principal managers, who are important intermediaries for large systems like City Schools, which has well over 150 principals.
Improvement science teams are most successful when integrated with everyday work and structures. City Schools’ Continuous Improvement Program Director, Amiee Winchester, finds that, “when you make improvement a thing that’s bigger and separate, there’s no time for it.” Improvement Coordinator Melissa Loftus notes, “we’re trying to align [efforts] with existing meetings. Coaches meet every Friday, so we’re asking questions like, ‘can we do PDSA [plan-do-study-act] cycles in those meetings?’”
3. Timing: Staggering the introduction of implementation and improvement science can maximize the impact of each approach.
Implementation science suggests a dynamic, four-phased approach: exploration, preparation, initial implementation, and full implementation. Initially, City Schools’ approach to introducing Eureka Math did not reflect lessons from implementation science and the district faced some challenges. In partnership with Leading Educators, City Schools changed to an approach more aligned to the dynamic phases: they first identified three pilot schools and designed professional learning (preparation), then rolled out supports to pilots and collected data (initial implementation), and next planned how to expand to eight more schools (moving to full implementation).
Improvement science applied to a new curriculum often works best once a district has achieved initial implementation. One challenge in implementing Wit & Wisdom was that the district introduced the curriculum and improvement science to teachers simultaneously. As Loftus reflected, the first year “was not the time to introduce improvement work. Year 1 is for learning the curriculum and doing it as best you can. Year 2 is the time to start figuring out what works and what doesn’t.” As teachers have become more comfortable with the curriculum, though, improvement science has helped them support students often marginalized in traditional education systems and respond to COVID.
4. External partners: Communicative and coherent partnerships are key to getting the most out of improvement and implementation science.
City Schools’ experience suggests that partners in implementation and improvement work require some distinct capabilities and relationships. Implementation intermediaries need relationships with leaders and the ability to use system-wide tools like strategic finance. Improvement partners emphasize in-school relationships and building capacity for ongoing refinement.
Both implementation and improvement, however, benefit from partner coordination. This coordination has at times been difficult for City Schools, as its ELA partners alone included a range of organizations and individual consultants. To build coherence, the district retained a coordinator to manage cross-partner collaboration.
One of education’s evergreen questions is how to ensure good ideas and innovations translate to impactful change – especially for students historically marginalized in education systems.
This case study looks at two of the leading methodologies for driving that change – implementation science and improvement science – through the experience of Baltimore City Public Schools (hereafter “City Schools”), which has achieved impressive progress in students’ outcomes in just a few years. City Schools has been deploying the lessons of both approaches - albeit less formally with implementation science than with improvement science - in its use of math and ELA curricula.
City Schools’ experience helps illustrate how improvement science and implementation science can serve complementary and interwoven purposes that together drive district-wide transformative change.
Definitions, intersections, and complementarity: Two interconnected approaches can work together for instructional transformation
A core priority within K-12 education is identifying how to transition promising practices from ideas to outcomes. To address this challenge, City Schools has been learning and drawing on lessons from two complementary approaches to change:
- Implementation science, a set of approaches that help institutions plan for, adopt, adapt, and sustain evidence-based solutions; and
- Improvement science, an approach to problem-solving focused on “rapid tests of change to guide the development, revision and continued fine-tuning of new tools, processes, work roles and relationships” (Bryk et al., 2015).
Implementation and improvement sciences are similar in that they both are data-driven approaches to shift a system from the status quo to more effective practices and reduce variation in outcomes. Both approaches require altering existing patterns of action and incorporate elements of behavior change to equip stakeholders to adopt new practices and lead change. Likewise, both approaches rely on data from research, real-time observation, and stakeholder experience to ensure the highest likelihood of success.
Yet they also have important distinctions (Figure 1). Implementation science often focuses on putting a solution into effect across an entire system, simultaneously affecting practitioners in many roles (e.g., from the central office through to individual classrooms). Because these changes generally involve novel practices across a wide-reaching scope and set of stakeholders, implementation science often approaches change through a multi-phased rollout process.
By contrast, improvement science takes an evolutionary, problem-focused approach that focuses on testing discrete changes to improve reliability and solve specific challenges with a practice or solution. Since improvement science takes a problem-solving approach to change practice, it often starts in a more bounded context where the problem is tangible (e.g., one school or set of classrooms) and accumulates insights for that context, some of which may ultimately scale more broadly. And rather than using a dynamic multi-phased process over a longer timeframe, improvement science emphasizes ongoing, rapid-cycle iteration.
Deployed strategically, the two approaches can be woven together for greater impact than each achieves on its own. City Schools’ experience learning and applying the lessons from each field - both formally and informally - helps illustrate that synergy, and the challenges to it, in practice.
City Schools is among the 50 largest districts in the nation, with one of the highest shares of students who are Black, Latino, experiencing poverty, and EL-designated. The district serves approximately 79,000 students across 168 schools and programs, spanning elementary through high school. The student body is roughly 77% Black, 14% Latino, an estimated 80% experiencing poverty, and 9% English-Learned-designated.
As in many large districts, City Schools has experienced several superintendent changes in the past decade. However, the current CEO, Dr. Sonja Santelises, served as City Schools’ Chief Academic Officer under a previous CEO (Dr. Andrés Alonso), providing “consistency in the message that having a clear vision around instruction is crucial,” explains Janise Lane, Executive Director of Teaching and Learning.
This vision led the district to engage in intensive implementation and improvement work over the past few years. This work included improvement science efforts in the secondary school grade band for the ELA curriculum introduced in 2018-2019 (Wit & Wisdom), and implementation of a new math curriculum (Eureka Math) and associated PL in 2017-2018. While the district has not yet applied implementation science as formally as improvement science, it has been finding its way to the lessons of implementation science in introducing both curricula. Its successes and struggles (discussed later on) both are illustrative of those lessons.
Improvement and implementation work are young, but the district is seeing encouraging improvements in student learning relative to 2015, before the curricula were adopted (Figure 2). Over the last five years, City Schools “has seen larger jumps in test scores than many school systems in the state. Scores have risen for two years in a row, while scores in Baltimore County have stagnated or dropped” (Bowie, 2020). Test score gains have been particularly pronounced for students often marginalized by education systems, with proficiency rates for Black students increasing an average of nearly five percentage points on some state tests and EL-designated students’ proficiency rates leaping by double digits in some cases. Even so, the district believes in the enormous potential of all students and recognizes these gains are just a start – a first step towards all students excelling - and need concerted effort to sustain amidst the challenges of 2020.
To learn how City Schools’ experience can inform districts’ use of implementation and improvement sciences, and to identify lessons for others, we interviewed over 20 City Schools staff and partners (listed in Appendix A) and paired what we heard with a research review.
Four categories of lessons emerged: the cultural emphases needed to spur change with each approach, the structures required to translate those cultural imperatives into action, the ideal timing over which to apply the approaches, and how best to leverage partners in doing so. The following sections address each category in turn.
Just as implementation and improvement science take deeply related but distinct approaches to transformation, they also work best under slightly different cultural environments – environments that leaders can help build.
Implementation: A widely shared sense of urgency makes implementing large-scale changes seem worthwhile and even welcome
Implementing a new solution, such as new instructional materials, often requires large changes across a district. This can be difficult and uncomfortable. As a result, implementation efforts are more likely to succeed if actors across the system share a felt need for change. According to Gustafson et al. (2013), “One of the key predictors of a successful change is whether the affected staff are dissatisfied with the current process…. Such tension should be an important consideration in deciding what projects to select.”
The Common Core State Standards helped create that sense of urgency in City Schools. The materials that the district used at the time did not meet the new standards, so teachers tried to design standards-aligned content from scratch. Not surprisingly, they often felt anxious about the lack of guidance in doing so, which created a widely felt need for new materials. According to Maggie Lasaga-Flister, a math teacher and coach, “The biggest shift for teachers was the Common Core State Standards and understanding expectations for students. Shifting to Eureka Math relieved anxiety about teaching to the standards.”
Yet districts do not have to – and often cannot – rely on the external context to create urgency. District leaders can take an active role in creating a recognition of the need for change. Heifetz, Kania, and Kramer (2004) note that, for leaders introducing a major change, “Harnessing [a] sense of disequilibrium – and making sure it stays productive – is a critical task.” Educational leaders advancing a new instructional system can actively cultivate a sense of disequilibrium to create momentum for change, such as by proactively communicating new materials’ role in the district’s vision for all students’ success.
In practice, urgency for change often starts with a clear, evidence-based, and shared vision for high-quality instruction. A strong vision can both build tension and guide its resolution through “schools and school district central offices working together to craft or continually negotiate the ﬁt between external demands and schools’ own goals and strategies” (Honig and Hatch, 2004).
City Schools is guided by a vision called its “Blueprint for Success.” In becoming the district’s CEO, Dr. Santelises interviewed students, families, teachers, and staff about how to improve outcomes for students and schools. Through this process, the district landed on three pillars of success: wholeness, literacy, and leadership (BCPS, n.d.).
City Schools’ leaders have used this vision to build energy for change by helping educators see gaps in current approaches. Janise Lane, Executive Director of Teaching and Learning, saw that “having an articulated vision that starts both CEO-down and teacher-up is critical…. We knew that our [ELA] materials weren’t building knowledge, so we shared resources about why building knowledge is foundational.” As always, there is some variation in teachers’ beliefs about the newer curriculum, but overall, because “we were planting the seed, people were much more ready to take up the curriculum rather than responding, ‘what in the world?’”
This intentional communication from district leaders helped educators appreciate the need for change and made the discomfort of a new instructional system seem worthwhile. Patricia Burrell, Principal at North Bend Elementary/Middle School, explains, “The messaging from Dr. Santelises is super important, starting with her vision for schools. Knowing your leader’s mission, goals, and the data behind it certainly helps in implementation …. It helped to understand why curricula were chosen, the tools that would be used to roll it out, and the supports we’d be given along the way. It wasn’t just dropped on us.”
Improvement: A sustained commitment to learning and growing can turn evolutionary changes into district-wide progress
While implementation often requires broad-scope changes, improvement science can be especially useful in contexts where the building blocks feel solid, but there is desire and energy for refinement. As noted by management scholars, “becoming a learning and continuously improving company demands more than debate and resources; it requires an organizational culture that constantly guides organizational members to strive for continuous improvement” (Ahmed, Loh & Zairi, 1999). For example, the automobile company Toyota – well known for its emphasis on improvement – has a culture such that “as soon as you start making a lot of progress toward a goal, the goal is changed and the carrot is moved. It’s a deep part of the culture to create new challenges constantly and not to rest when you meet old ones” (Sturdevant, 2014).
City Schools’ leadership has invested in building a culture committed to learning and refinement. Anthony Bryk, President of the Carnegie Foundation for the Advancement of Teaching, noted that he is “very impressed with Baltimore and their commitment to make improvement science work in a thinly resourced district (financially)…. They live a belief that ‘we can and we should do much better.’” Elizabeth Manolis of Great Minds (Wit & Wisdom’s publisher) shares this sentiment: “I love that [when the district hits a roadblock] there is this mindset of, ‘We’re not quite there. That’s okay. We’re not stopping, so what are we going to try differently?’” School leaders also see commitment to learning as crucial to City Schools’ improvement efforts: “Our school is a place where teachers want to learn and grow. That was a key piece for improvement” efforts, explains Principal Burrell.
The district is channeling this culture into a set of ongoing teacher-led efforts to refine how the ELA and math curricula are taught, with the support of capacity-building from the Carnegie Foundation and others. Teacher-leaders learn, and then engage their peers in, improvement science tools, such as empathy interviews (semi-structured interviews protocol to uncover hidden needs) and disciplined inquiry cycles (structured tests of small-scale changes) to identify problems and ideas from educators enacting the curriculum in classrooms. Patience Hein, an Assistant Principal at North Bend, has found that the improvement work with the school’s principal and literacy coach “helps us identify small changes in instruction that might generate large results.”
With keystones such as high-quality curricula and aligned PL in place, City Schools’ commitment to constant learning supports improvement science practices that elevate students’ outcomes.
With those cultural foundations in place, district leaders and their partners can consider the structures necessary to support change. Whether for implementation or improvement, it is now a widely agreed best practice to create dedicated teams that enable communication and coherence across organizational silos. These structures support change by creating clear communication pathways that, in turn, “enable people to access, share, make meaning of, and use knowledge to problem solve” (Farrell, Coburn, and Chong, 2017).
Baltimore’s experience and related research suggest both commonalities and differences in team structures across key stakeholder groups, as detailed further below.
Implementation and improvement: Both approaches benefit from teacher networks and engagement with students, their families, and the community
Effective implementation and improvement share the feature that teacher support and peer networks are crucial to success. Teachers are vital agents of coherence, and like most of us, place special credibility in their peers. Investing in teacher-leaders and being attentive to peer networks and “opinion leaders” can foster uptake of implementation priorities and the spread of tested ideas from improvement science. Spillane and Hopkins (2018), for instance, confirmed that “teachers’ interactions with peers about mathematics instruction were associated with changes in their beliefs over time.” As such, they emphasize avenues to elevate key perspectives and support “teachers in developing knowledge specific to implementing reform-oriented instruction.”
City Schools, similarly, leveraged peer relationships to build buy-in for implementation of Wit & Wisdom by investing deeply in teachers as experts: “We had teacher-fellows built into our program,” explains Janise Lane. “These teachers got additional training from Great Minds to be experts. And, thank goodness for them, they started tweeting about what was happening in their classrooms. They started advertising what kids in Baltimore deserve.”
Likewise, City Schools has facilitated improvement partly by creating structures for peer sharing and collaboration. In fact, though the improvement science support focuses on Wit & Wisdom, the district is using similar structures for math. In collaborative work time, says one math teacher-leader, Maggie Lasaga-Flister, teachers figure out “what’s working with one set of kids and not others…and take that to the [cross-school] network meeting to share.” Others echo this sentiment: “At the network half-day professional development meetings, which are mostly teacher-led…we strategize, plan, work collaboratively for upcoming lessons and analyze common assessments,” and problem-solve for improvement, notes Emmanuel Ramos, a high-school math teacher-leader.
Additionally, engaging students, their families, and the community is vital for short- and longer-term success. According to the National Implementation Research Network, core competencies of implementation partners include the ability to facilitate co-creation with a variety of stakeholders by building bridging relationships, addressing power differentials, learning from stakeholders, and leading collaborative design processes (Metz, Louison, Ward, & Burke, 2017).
City Schools built and realized its vision with student, family, and community voice. Even with conviction that a high-quality curriculum is crucial, City Schools’ CEO, Dr. Sonja Santelises, says that she “can’t presume to know what people want. I am able to make my case only by first demonstrating that I actually care about what families think.” Consequently, when developing their instructional vision, City Schools held 11 separate community events with hundreds of parents, educators, and other stakeholders to identify priorities across its over 150 schools. City Schools is now building on that engagement with students, families, and communities to facilitate COVID response, including by hosting virtual events for students to help guide the response strategy.
One priority that emerged repeatedly was the need for cultural relevance. When adopting a new curriculum shortly after Freddie Gray’s death, Dr. Santelises recognized that “in a community with open gaping wounds that were local and national, there was no way we were going to brush it aside by Wit & Wisdom just being more culturally relevant than what came before.”
Hearing this feedback, City Schools took on a set of mutually reinforcing implementation priorities to support student success. As the district implemented Wit & Wisdom, it worked with Great Minds to host events to discuss culturally relevant texts and instruction. Meanwhile, the district launched a complementary effort called BMore Me, a teacher-led social studies program that aligns state standards with the stories and histories of students in Baltimore.
In retrospect, Dr. Santelises reflects that incorporating families’ feedback by implementing BMore Me alongside Wit & Wisdom “absolutely has been beneficial...The schools that didn’t have Wit & Wisdom and opted not to have BMore Me are now the ones that face vociferous protest against the lack of culturally relevant curricula in their school...” For schools that included families’ priorities, the engagement around cultural relevance with Wit & Wisdom and BMore Me is now “proving to be a big support in this time of racial justice uprising.”
Implementation: New instructional systems often require changes to teams and roles to facilitate coherence across many schools
Given the extensive change implementation entails – and the need to deprioritize other efforts temporarily to enable focus – it is often helpful to craft new or revised structures and roles. In implementing Wit & Wisdom, for example, the districts’ leadership - especially the former Chief Academic Officer (CAO) and Chief of Schools Officer (COSO) - had to rework their previously distinct spheres of operation to coordinate closely. Maggie Slye of Leading Educators noticed the payoff of this coordination when she later stepped into the district: in many districts “there can be tension between the academic office and the schools office,” but in Baltimore the former CAO and current COSO “prioritized acting as though they were one person.” Within the last year, City Schools’ CAO position has vacated and the COSO also has served as the Acting Chief Academic Officer, helping ensure continued coordination between the offices (Figure 3).
Likewise, at the school level, teachers’ peer coaches often provided curriculum support without turning to district-level Academic Content Liaisons, who were supposed to serve school leaders as content experts. Leading Educators set the norm that these roles collaborate in planning PL for Eureka Math – “the first time I’ve seen [district-based] Academic Content Liaisons and [school-based] Academic Planning Facilitators co-planning and co-facilitating Professional Learning Communities for their teachers in the decade since the role’s creation,” Slye reports hearing from a district leader.
Perhaps no role definition is more important than that of principals and principal supervisors. Interviewees consistently emphasized that principals are essential drivers of implementation coherence and improvement success. For example, Nadya Chinoy Dabby, who oversaw the federal Investing in Innovation (i3) program and is now Chief Growth Officer at Pivot Learning, noted that “in i3, there was a tendency to focus on leadership at the district level,” but “changes that endured often got translated and held onto through principals.” According to Coburn (2005), “Principals influence teachers’ enactment by shaping access to policy ideas…and creating substantively different conditions for teacher learning in schools.”
At the scale of a district like City Schools, which has well over 150 principals, intermediary structures like principal supervisors are especially important in implementation. In fact, in a 2014 study of governance in five districts, Johnson et al. found that “each of the districts we studied had created a role for intermediaries, district administrators who coordinated work with the schools. Intermediaries were agents of coherence.”
City Schools’ intermediary structure, Community Learning Networks, bridges principals to the central office and supports them in leading implementation at their schools. Principals, such as Patricia Burrell at North Bend, found it “great to have experts through the CLN” to “encourage sharing best practices and extended planning for schools in our network” during implementation of Wit & Wisdom. Similarly, as the district transitions from implementation to improvement of Wit & Wisdom, external partner JB Buxton named “Baltimore’s principal supervisors [as] key players that created alignment across roles.”
Improvement: Improvement science thrives when integrated into existing systems
Whereas implementation often requires substantial revising and restructuring of teams and roles, improvement science is most successful when integrated with existing structures - albeit with the same data-driven discipline as in implementation - so users can incorporate it into their daily work. City Schools’ Continuous Improvement Program Director, Amiee Winchester, for instance, has found that it is important “not to make improvement feel like it’s something separate…. When you make it a thing that’s bigger and separate, there’s no time for it.”
City Schools, similarly, has learned that folding improvement science into existing structures, such as by including it in school teams’ regular meetings, facilitates its adoption. Initially, improvement science work “involved separate meetings with coaches,” says Continuous Improvement Coordinator Melissa Loftus, “but now, we’re trying to align with existing meetings. Coaches meet every Friday, so we’re asking questions like, ‘can we do Plan-Do-Study-Act cycles [a common improvement science tool] in those meetings?’ We can take advantage of existing things.”
To this end, improvement science relies especially strongly on school-level experimentation and networking. For instance City Schools partnered with the Carnegie Foundation and Bill & Melinda Gates Foundation - which have been exploring and strengthening schools’ use of improvement science - to support a “Networked Improvement Community” of 26 middle and high schools across the district. In this way, the district approaches improvement science “with the school as the unit of change,” in which “school staff [engage] in intensive coaching and data review sessions with coaches who are both content and instructional experts” (Carnegie, “BCPS”). Teacher-leaders known as Literacy Improvement Fellows receive training in how to apply improvement science to their classroom teaching, with the aim of spreading this capacity to peers in their building.
Improvement leaders like Winchester, Loftus, and Director of Strategy & Continuous Improvement Sarah Heaton then help ensure “social learning across school contexts,” notes Heaton. One objective of her team is to build a broad “continuous improvement” approach through the district that can draw on lessons from implementation science to scale innovations that emerge from school-based improvement science efforts.
This distributed leadership has proven particularly valuable during the COVID-19 pandemic. For instance, when transitioning to distance learning, Emily Jaskowski, a Literacy Improvement Fellow, helped colleagues use improvement science “to increase attendance in online sessions. We came up with a list of four or five ideas. Now, we’re testing them one at a time to see what’s most effective” methodically, instead of “throwing everything at the wall.” School-level leaders can help their peers and principals make data-driven improvements, building on structures with a longer history that provide continuity for refinement even amidst a crisis.
Implementation and improvement science are complementary approaches and work best when used as such. In particular, staggering curriculum implementation and the formal introduction of improvement science can maximize the impact of each effort. Because of the transformative nature of implementing a new curriculum, it often makes sense to start by ensuring teachers build a foundational understanding of the materials before applying formal improvement practices.
Implementation: Implementation science suggests a dynamic, four-phase approach that can help districts plan their rollout process
When a district embarks on extensive change, implementation science can help structure its efforts. Many implementation scholars outline four iterative phases: exploration, preparation, initial implementation, and full implementation, each of which has a key milestone (Figure 4).
Though City Schools did not explicitly deploy implementation science in rolling out Eureka Math, early learning led the district to a phased and dynamic approach more consistent with the science. The district started initial implementation in the 2017-2018 school year but quickly found that support for curriculum implementation was a top need cited by teachers. Consequently, City Schools collaborated with Leading Educators to revisit and revise the implementation plan, reflecting the research finding that the “stages are not linear…, [and] there are times when an organization will move among stages due to changes in staff, funding, leadership, or unsuccessful attempts at employing the innovation with high fidelity,” as indicated in Figure 4 (SISEP, n.d.).
In particular, in their partnership with Leading Educators, the district developed a phased implementation process more aligned to preparation, initial implementation, and full implementation: The partnership first identified three pilot schools and designed professional learning (preparation), then rolled supports out to these pilot schools and collected data on their experiences and outcomes (initial implementation), and finally developed a plan to expand to eight more schools and build scaling capacity for broader scaling (moving towards full implementation).
Improvement: Improvement science works best once a district has achieved initial implementation of a curriculum, so as to manage cognitive load
Once a district begins implementing a curriculum, improvement science is a powerful tool to test adaptations or new practices that can refine and sustain the curriculum’s use. Although improvement science components may be part of initial implementation (e.g., piloting a curriculum to troubleshoot challenges early), it is valuable for educators to have a deep understanding of the curriculum before districts roll out formal improvement science efforts.
One challenge in City Schools’ implementation of Wit & Wisdom was that the district introduced the new curriculum and improvement science simultaneously to teachers. Theodora Johnson, a teacher and Literacy Improvement Coach, remarked that, while adjusting to Wit & Wisdom, “none of us on any level were comfortable enough with the curriculum to plan or know what the struggles would be or where to use” some of the formal improvement science tools.
Johnson’s experience is validated by a body of evidence finding that “adaptations made after a model had been implemented with fidelity were more successful than modifications made before full implementation” (Fixsen et al., 2005). Recent experimental research, similarly, found that teachers who received fidelity-focused PL in year 1 followed by adaptation-focused PL in year 2 were the only group with statistically significant professional learning gains and improvements in their students’ learning outcomes (see Figure 5; Quinn and Kim, 2017).
City Schools independently arrived at the same conclusion as Quinn and Kim. As Continuous Improvement Coordinator Melissa Loftus reflected, the first year of use of Wit & Wisdom “was not the ideal time to introduce improvement work. Year 1 is for learning the curriculum and doing it as best you can. Year 2 is the time to start figuring out what works and what doesn’t.” Beth Sappe, City Schools’ Director of Mathematics, is applying this lesson to Eureka Math: “we would not have been ready for content cycles in year one;” instead, “Leading Educators is helping us deepen our content knowledge and create a foundation for high-quality professional learning” and improvement.
As teachers have become more comfortable with the curriculum, though, improvement science has helped them identify ways to support students with the most need. One middle school teacher “tested three different change ideas [and] the results from the first tests were informative and impressive…For example, students almost passing phonics benefited from fluency passages. But those who needed support didn’t.” Teachers’ comfort with improvement science is also helping them to make smart adaptations to COVID’s challenges. JB Buxton, who helped coordinate external partners for City Schools, posits that “the fact that they’ve started to develop improvement chops as they move into the COVID reality of uncertainty [means] they have a shot to learn and pivot in a way that a lot of districts won’t be thinking about. A lot of districts will just be surviving; Baltimore can be learning.”
To manage curriculum implementation, improvement, and many other district systems, Baltimore brought in several external partners with skillsets specific to each approach. While implementation and improvement partners may require somewhat different skills, there is also substantial overlap - and what is most important is ensuring coherence across the multiple organizations with which a large district is likely to partner.
City Schools’ experience suggests that implementation and improvement partners require some distinct capabilities and relationships:
- Implementation partners need strong relationships with district leaders and the ability to make strategic use of district-wide tools. Elizabeth Manolis, Director of Humanities Implementation Success at Great Minds, states that “a regular, intimate relationship with district leadership is critical because they have insights into the daily successes and challenges teachers face with early implementation.” This relationship allows implementation partners to attend to system-wide changes (e.g., in the district’s approach to PL). To that end, implementation partners must also be able to help districts leverage system-wide tools, including strategic resource allocation and revision of master schedules.
Moreover, even as implementation partners’ direct relationships with the central office are crucial, they work best when partners coordinate with one another. As Executive Director of Teacher and Learning, Janise Lane, recounts, “one component that mattered a lot was the coordination we had with JB Buxton. He would meet regularly with me and understand our focus and then play a pivotal role as a liaison between partners and district leadership.”
- Improvement partners often place more emphasis on in-school relationships and must be adept at building capacity for ongoing refinement. In the experience of Patience Hein, an Assistant Principal at North Bend, Carnegie works closely with individual improvement fellows: “Meetings were just improvement fellows with Carnegie – improvement fellows were really the ones implementing the change theory in schools.” Through in-school relationships, improvement science partners can support context-specific iteration to inform aggregate district-wide lessons.
At the same time, the two approaches also share similarities in the attributes of a successful partnership
Implementation and improvement: Partners must work together effectively to maximize impact on student learning outcomes
Creating coherence across all partners has been difficult for City Schools. In ELA alone, City Schools’ external partners included Great Minds, New Teacher Center, Carnegie, and individual consultants (Figure 6). In one improvement fellow’s experience, “the literacy coach was engaged on improvement science but they sometimes got mixed messages on how to use it with Wit & Wisdom,” including advice from partners that “didn’t always align with” improvement science.
To help build this coherence, the district brought in an external consultant to establish a structure for regular cross-partner collaboration. JB Buxton, the consultant, recognized that many of the providers “were going into schools and doing learning walks and providing feedback. They were inundating [teachers] and were not aligned.” Consequently, Buxton focused “a lot of our early work on clarifying the partners’ roles, responsibilities, and relationships.” According to Anthony Bryk of the Carnegie Foundation, this challenge stems from the fact that “district systems are designed to add things, not to create internal coherence.” Bryk noted that a key component of Buxton’s role involved “asking the question, ‘What are you going to take off the table?’” Buxton regularly checked in with district leadership, improvement coordinators, and all partners, and facilitated quarterly cross-partner meetings with district leadership to help mitigate coordination challenges and explore possible causes of variation in practice and outcomes.
As City Schools’ experience shows, coherence even across partners committed to collaboration should not be taken for granted. Districts (with support from funders) need to take responsibility for ensuring partners align with each other and the district’s vision for instruction, and to ensure that the district is truly leading the work. Buxton saw that “the district didn’t want [improvement efforts] to be just another initiative or siloed.” These priorities led to meetings that, according to Great Minds, “always end up pushing the district and partners to think differently and hastens the pace of improvement.” As districts and funders consider strategic partnerships for implementation and improvement, regular cross-partner and cross-district communication helps create coherence, accelerate district transformation, and keep the focus on in-district leadership and capacity.
City Schools’ experience demonstrates how aligning implementation and improvement science can drive progress in students’ outcomes - especially among students who are Black, Latino, experiencing poverty, and EL-designated. Moreover, there is much more to learn about how to make the approaches even more effective. For instance, as City Schools continues to use these practices, it can explore two nascent areas in applying implementation and improvement science in education:
- Increasing student, family, and community participation in implementation, improvement efforts, and cross-district coherence. In adopting Wit & Wisdom, City Schools engaged students, families, and community stakeholders through public feedback sessions, a student-facilitated Facebook Live Q&A with district leadership, and opportunities for parents to interact with potential materials (EdReports, “Redefining Engagement”). Similarly, with CI, teachers regularly seek student perspectives through empathy interviews to understand the instructional system from their viewpoint as a way to improve it. City Schools’ new social studies program, BMore Me, is at the leading edge of end-user engagement and is piloting an innovative assessment tool to help teachers better understand how students experience and engage with the curriculum—by asking students directly.
Moving forward, the district can explore further avenues to elevate student and community leadership in mathematics and ELA implementation and improvement, such as by including parents and community leaders on implementation teams or training students as co-designers for inquiry cycles. In fact, City Schools already is exploring how to build on BMore Me to increase student leadership and voice throughout the district.
- Using improvement science to identify the core elements of a curriculum versus those that are adaptable to context can help a district scale a curriculum. A leading framework for health implementation research calls for distinguishing core components (“the essential and indispensable elements of the intervention”) from an adaptable periphery (the “adaptable elements, structures, and systems related to the intervention and organization into which it is being implemented”; Damschroder et al., 2009). Now that City Schools is well into the implementation process in two evidence-based curricula, it can use improvement science tools such as Plan-Do-Study-Act cycles to test how adaptations to the implemented curriculum affect outcomes. Articulating the core versus adaptable elements of the curriculum and aligned supports can help scale best practices for each curriculum while identifying elements for which exact implementation is less crucial.
Even as City Schools continues to learn from implementation and improvement science, its application of these two approaches already has driven impressive progress for students often marginalized within education systems. Districts, and funders looking to support them, can use these lessons from Baltimore City Public Schools to align implementation and improvement sciences strategically to transform student learning outcomes.
- Ofelia Arcillo, Math lead
- Anthony Bryk, President, Carnegie Foundation for the Advancement of Teaching
- Patricia Burrell, Principal, North Bend
- JB Buxton, Former consulting partner and President of Durham Technical Community College
- Nadya Chinoy Dabby, Chief Growth Officer, Pivot Learning
- Chong-Hao Fu, Leading Educators
- Sarah Heaton, Director, Strategy & Continuous Improvement
- Patience Hein, Assistant Principal & 2019-20 Improvement Fellow, North Bend
- Emily Jaskowski, ELA teacher & 2019-20 Improvement Fellow
- Theodora Johnson, ELA teacher & 2019-20 Improvement Fellow
- Janise Lane, Executive Director of Teaching and Learning
- Maggie Lasaga-Flister, Math lead
- Melissa Loftus, Continuous Improvement Coordinator
- Elizabeth Manolis, Director of Humanities Implementation Success, Great Minds
- Laura Minicucci, Blueprint Literacy Coach, North Bend
- Emmanuel Ramos, Math lead
- Christina Ross, Program Manager for Blueprint Initiatives
- Sonja Santelises, CEO, Baltimore City Schools
- Beth Sappe, Director of Mathematics
- Maggie Slye, Managing Director of Networks (formerly, Managing Director of Thought Leadership), Leading Educators
- Amiee Winchester, Continuous Improvement Program Director
- Anonymous middle school teacher & 2019-20 Improvement Fellow
Ahmed, P. K., Loh, A. Y., & Zairi, M. (1999). Cultures for continuous improvement and learning. Total Quality Management, 10(4-5), 426-434.
Bhuiyan, N., & Baghel, A. (2005). An overview of continuous improvement: from the past to the present. Management decision.
Baltimore City Public Schools. (2020). City Schools at a glance. Retrieved from: https://www.baltimorecityschools.org/district-overview
Baltimore City Public Schools. (n.d.). Building a Generation: City Schools’ Blueprint for Success. Retrieved from: https://www.baltimorecityschools.org/sites/default/files/2019-01/Blueprint-complete.pdf
Bowie, L. (2020, February 11). Baltimore City school board gives chief officer Sonja Santelises rare second contract. Retrieved from: https://www.baltimoresun.com/education/bs-md-ci-santelises-contract-renewal-20200211-ae44mc5unrfetfjy3vsmgsn5la-story.html
Bryk, A. S., Gomez, L. M., Grunow, A., & LeMahieu, P. G. (2015). Learning to improve: How America’s schools can get better at getting better. Harvard Educational Review. Retrieved from: https://www.hepg.org/her-home/issues/harvard-educational-review-volume-85-number-4/herbooknote/learning-to-improve
Carnegie Foundation. (n.d.). Baltimore City Public Schools. Retrieved from: https://www.carnegiefoundation.org/our-work/baltimore-city-public-schools/
Carnegie Foundation. (n.d.). Learning to Improve Glossary. Retrieved from: https://www.carnegiefoundation.org/resources/learning-to-improve-glossary/#quality
Coburn, C. (2005). Shaping Teacher Sensemaking: School Leaders and the Enactment of Reading Policy. Educational Policy, 19(3), 476.
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation science, 4(1), 50.
Farrell, C. C., Coburn, C. E., & Chong, S. (2019). Under what conditions do school districts learn from external partners? The role of absorptive capacity. American Educational Research Journal, 56(3), 955-994.
Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., Wallace, F., Burns, B., ... & Chambers, D. (2005). Implementation research: A synthesis of the literature.
Gustafson, D. H., Sainfort, F., Eichler, M., Adams, L., Bisognano, M., & Steudel, H. (2003). Developing and testing a model to predict outcomes of organizational change. Health services research, 38(2), 751-776.
Hamm, R. (2020). An Example at the Intersection of Obstetrics and Implementation Science. Presentation, 40th annual pregnancy meeting, Society for Maternal-Fetal Medicine.
Heifetz, Ronald A., John V. Kania, and Mark R. Kramer. (2004). Leading boldly. Stanford Social Innovation Review. Retrieved from: https://ssir.org/articles/entry/leading_boldly
Honig, M. I., & Hatch, T. C. (2004). Crafting coherence: How schools strategically manage multiple, external demands. Educational Researcher, 33(8), 16-30.
Hopkins, M., Spillane, J. P., Jakopovic, P., & Heaton, R. M. (2013). Infrastructure redesign and instructional reform in mathematics: Formal structure and teacher leadership. The elementary school journal, 114(2), 200-224.
Hopkins, M., & Woulfin, S. L. (2015). School system (re) design: Developing educational infrastructures to support school leadership and teaching practice. Journal of Educational Change, 16(4), 371-377.
Horner, Blitz, Ross 2014; Monzalve, 2016; Dymnicki et al., 2019; National Academies of Sciences, Engineering and Medicine, 2019.
Johnson, S. M., Marietta, G., Higgins, M. C., Mapp, K. L., & Grossman, A. S. (2014). Achieving coherence in district improvement: Managing the relationship between the central office and schools. Harvard Education Press.
Metz, A., Louison, L., Ward, C., & Burke, K. (2017). Implementation Support Practitioner Profile: Guiding principles and core competencies for implementation practice. Chapel Hill, NC: National Implementation Research Network, University of North Carolina at Chapel Hill.
Park, S., Hironaka, S., Carver, P., & Nordstrum, L. (2013). Continuous Improvement in Education. Advancing Teaching--Improving Learning. White Paper. Carnegie Foundation for the Advancement of Teaching.
Quinn, D. M., & Kim, J. S. (2017). Scaffolding fidelity and adaptation in educational program implementation: Experimental evidence from a literacy intervention. American Educational Research Journal, 54(6), 1187-1220.
Riddle, W. (2015). Implications of community eligibility for the education of disadvantaged students under Title I. Center on Budget and Policy Priorities. Retrieved from: https://www.cbpp.org/research/food-assistance/implications-of-community-eligibility-for-the-education-of-disadvantaged
SISEP. (n.d.). Topic 1: Implementation Stages Overview. Active Implementation Hub. Retrieved from: https://nirn.fpg.unc.edu/module-4/topic-1-implementation-stages-overview
Spillane, J. P., Hopkins, M., & Sweet, T. M. (2018). School district educational infrastructure and change at scale: Teacher peer interactions and their beliefs about mathematics instruction. American educational research journal, 55(3), 532-571.
Sturdevant, D. (2014, February 1). (Still) Learning from Toyota. McKinsey Quarterly. Retrieved from: https://www.mckinsey.com/industries/automotive-and-assembly/our-insights/still-learning-from-toyota#
Jeremy Avins is an Associate Principal at Redstone and leads much of the firm’s work on economic well-being and mobility. His work includes supporting the development of philanthropic education strategies that support district-provider partnerships to apply evidence and research to district-wide implementation of coherent instructional systems.
Delaney Overton is an Analyst at Redstone. Delaney supports nonprofit and philanthropic strategy development in education, economic justice, and human development. Her work has included designing strategies for innovative youth programs, teaching K-3rd academic enrichment classes, and conducting research on news and media policy.
Nathan Huttner is Redstone’s Managing Director and leads the firm’s education practice. Nathan has led work to design and evaluate a wide range of innovative education strategies in K-12 and higher education for large foundations and leading NGOs. His work focuses on how policy, practice, and markets interact to enable philanthropic impact.
The authors would like to thank the interviewees (listed in Interviewees & References) for contributing their time and experiences, the Bill & Melinda Gates Foundation for supporting the case study, and Teresa Rivero, Rachel Leifer, and Vivian Mihalakis. The findings and conclusions contained within are those of the authors and do not necessarily reflect positions or policies of the Bill & Melinda Gates Foundation.