Can Early Teacher Evaluation Findings Help Change the Debate?

Over the past few years, states and school districts across the country have devoted significant resources to the design and roll-out of new teacher evaluation systems.  Driven at least in part by requirements attached to Race to the Top funding, the new systems have inspired heated debate over the efficacy of factoring student achievement data into a teacher’s performance assessment. The New York Times recently shared some initial findings from states that have launched new evaluation models including Michigan, Florida and Tennessee, reporting that the vast majority of teachers- upwards of 95 percent in all three- were rated as effective or highly effective. Although the analysis of these numbers has only just begun, the Times reports that some proponents of the new evaluation models admit that the early findings are “worrisome”.  And even though it is still early, we can reasonably anticipate that if the trend continues- and the findings from the new evaluation systems reveal no significant departure from more traditional methods of evaluation- we may start to have a lot more people looking at the complicated data analysis driving teacher evaluation systems linked to student achievement data and asking “what’s the point?”

It’s a good question, really, and one that probably hasn’t gotten enough thoughtful attention in the midst of the controversy surrounding them: What is the point of linking student achievement data to teacher evaluations?  Should we take it for granted that a primary goal- if not the primary goal- of these efforts is to identify and eliminate bad teachers?  If this is the case then these early findings should be a cause for concern, especially given the time and money being spent to collect and analyze the data.  If replacing bad teachers with good ones is the magic bullet for public education reform, it will take a pretty long time at this rate.

Of course, even opponents of the new evaluation systems would probably admit that the magic bullet theory is an oversimplification. Furthermore, it’s much too early to look at these numbers and extrapolate any meaningful conclusions about the actual number of ineffective teachers or even the accuracy of the evaluations themselves. Hopefully what these findings might do is allow us to finally begin to broaden the scope of our national conversation about how the linkages between teachers and students could actually drive education reform.  States and school districts implementing new evaluation systems have tried with varying degrees of success to communicate the message that linking student achievement data to teacher practice isn’t just about punitive measures- it also has important implications for improving professional development and teacher preparation programs by identifying shared practice linked to positive student achievement and replicating those practices in classrooms across the country. But that message is often overshadowed by the anxiety surrounding the punitive side of evaluation and underscored by public struggles with local teacher unions. If nothing else, these early findings might create an opening in the current debate for a more thoughtful discussion about the broader possibilities for linking teacher practice to student growth.

-Jacqueline Skapik

Leading Change in Education Reform Efforts

“A leader is best when people barely know he exists, when his work is done, his aim fulfilled, they will say: we did it ourselves.” —Lao Tzu

Leading people through a process of change is difficult, particularly in big, entrenched systems like traditional schools and districts. A school leader I know once told me that traditional school systems are like giant ocean tankers, you can make them change direction but it takes a lot of time and energy. Few school and district leaders are prepared to turn their ocean tankers around, particularly within the timelines and to the degree required for Race to the Top (RTT) to meet its ambitious goals.

As evidenced by the recent RTT year two reports released by US DOE, one of the greatest challenges to the success of the RTT reforms is not the content of the changes themselves, but simply that RTT entails significant change on the parts of individuals and systems, and change is hard.

Psychology research tells us that people don’t like change (status quo bias). Change takes effort, causes discomfort and sometimes can be downright painful. People fear the unknown. They wonder, “Will the work and the pain be worth the effort?”

Yet there is an urgent need for change in our education systems to ensure that all students are prepared for success in college, work and life. This need can be seen in student proficiency data from across the country. Rhode Island recently released the latest round of state assessment results, which were a grim reminder of how far we still need to go and how long it takes for systemic change to have an impact at the classroom level.

So how do we address the challenge of leading change?

In Rhode Island, one way we are supporting local leadership and spreading effective ideas to support RTT implementation is through the Collaborative Learning for Outcomes (CLO) process. Through the CLO process, Rhode Island district leadership teams meet regularly in facilitated sessions to share effective practices and learn from one another regarding RTT implementation strategies. The CLO process has provided a forum for district and school leaders from across the state to dig down into concrete strategies to support RTT implementation and to discuss mitigating the complex challenges they face on a daily basis with peers who struggle with the same issues.

In my work with the CLO teams in Rhode Island, the successful education leaders I have observed all share and act on the following beliefs about leadership:

(1) Communication must be a two-way street. To lead people into the unknown, you must listen, have honest dialogue, and be transparent about the work ahead. A number of district leaders who shared during CLO meetings that they created genuine opportunities for their teachers to express and receive answers to their concerns about the new RTT systems were the ones who were also most likely to report that everyone felt they were on the same team when it came time for implementation.

(2) Everyone must share ownership of the work. Distributing leadership responsibilities among those affected the most by changes builds internal champions and on-the-ground capacity, giving people responsibility leads to increased motivation to move the work forward, and getting implementer input on the “how” of the reform greatly increases the chances for success. Through the CLO process, I saw how school leaders who did not engage teacher-leaders in their buildings in developing implementation strategies were almost universally unable to move reform efforts forward with any reasonable speed. The opposite was true of those leaders who created real opportunities for teachers to hold responsibility for success.

(3) Leaders must support those on the front lines of change. Success depends on whether leaders can be creative about finding new resources and using existing resources, provide staff with needed training, and flexibly support staff to face the unknown. All educators in Rhode Island are working within the constraints of limited human and financial resources and the aspirational goals for RTT implementation. Many CLO discussions center on the challenge of stewarding resources wisely and creatively. While no one has found a magic bullet, those leaders who have acknowledged the insufficiency of the traditional structures for budgeting and using human capital are finding a variety of new ways to plan resource use so that they can provide as much support as possible for their staff.

By implementing innovative strategies, like the CLO process described here, we can help education leaders learn about and adopt practices that will increase the likelihood of success of RTT and other critical educational reform efforts.

–EFB

Do States Lack the Capacity for Reform?

Michael Usdan and Arthur Sheekey just wrote a great commentary on the complex and evolving relationship between federal policy, the State Education Agency, and the human capacity to get it all done.  In their essay, “States Lack the Capacity for Reform” over in Education Week, Usdan and Sheeky argue that, “In essence, most state education departments remain almost wholly owned federal subsidiaries, with well over half their budgets emanating from federal funds.”  Because of this, many states under-fund State Education Agencies (just as we have seen local governments under-fund their own school districts if the district is largely funded by the state—like here in Baltimore).  Take this on top of declining budgets and the huge push to reinvent state standards through the common core, implement new teacher evaluation systems, and develop new data tools, and you have a mountain to move.

Usdan and Sheeky point out the structural and organizational changes that Delaware and Tennessee are making in response to these pressures.  This is absolutely needed. But, I can’t help but think that the brand of the poor state education bureaucrat needs some scrubbing as well.  After all, the success or failure of all education reform today rests on the weary shoulders of a few talented managers in the states and districts taking them on.  These managers live and die by the axiom of what I call, “the burden of being useful” in districts and SEAs.  The “burden” afflicts talented managers who are found to possess the unique ability to carry water on difficult projects and deliver time and time again.  Drowning in complex new challenges, districts and SEAs not only give these people the hardest and most difficult projects, but every other project they can throw at them as well.    These stars burn bright, but they usually burn out in two to three years.  This has to change if we expect the current wave of reform to sustain.

If you have had the luck of working in an SEA or district taking on the reform challenge, you know it is mix of politics, bridge building, data crunching, sweat, organizational psychology, and managing a to-do list a mile long.  On the worst days, it feels like hell. But, by and large, shouldering the work of education reform to me feels like what I imagine it must have been like in Silicon Valley in the 80s.  We are writing history as we go.  And the possibility that we will build a fundamentally better system of education for our nation’s kids is before us.  If you are coming out of your MBA or MPA program, TFA or TNTP class, or are tired of your middle manager job in corporate America, this is the most exciting place to be in America.  And, your talents will grow substantially by pressing your shoulder against this plow.

To complete and sustain education reform, we need talented managers in School Districts and SEAs.  And to attract these talents to education and relieve the burden that they currently feel, we can start by recasting the story of what it means and what it is like to work for school districts and State Education Agencies (BR).

You Can’t Comply Your Way to Common Core Implementation

Forty-five states plus the District of Columbia and the U.S. Virgin Islands have adopted the new Common Core State Standards (CCSS). The CCSS, developed by a state-led initiative, are intended to align instructional expectations across the country so that all students receive a rigorous and relevant education that will prepare them to be successful in college and careers (http://www.corestandards.org).

Transitioning to the CCSS is presenting states with the daunting challenge of implementing the new standards in a context of high levels of public interest, high-profile funding opportunities (like Race to the Top) and new partnership opportunities with other states (like the new multi-state testing consortia).

Successfully transitioning to the CCSS will require districts and states to do work they have never done and to develop partnerships with each other that haven’t existed in the past. Historically, state education departments and local districts have had two types of relationships: (1) the state monitors for compliance and enforces state and federal laws and regulations, and (2) the state provides educational resources and/or technical assistance to local districts. These traditional modes of interaction are not sufficient to support the successful local implementation (with permeation down to the classroom level) of the ambitious reform agendas being taken on across the country.

In this context, how can state leadership ensure that the implementation of CCSS in their local districts is successful?

States must not only create new content and systems associated with transitioning to the CCSS as well as other reform efforts and monitor participation by local districts, but must actively provide a facilitated opportunity for district leadership teams to work together to strategize around significant implementation challenges, share best practices and develop cross-district relationships. They must also develop an in-depth understanding of state-wide implementation challenges and strengths, as well as pockets of weakness across the state, in order to tailor technical assistance to actual district needs.

We have worked with Rhode Island to develop a new process called Collaborative Learning for Outcomes (CLOs) that is helping the state do exactly this—redefine the state-district relationship, improve communication around implementation challenges and facilitate communities of practice for district leadership teams around reform implementation (including transition to the CCSS). We have found that the CLO process has enabled this work to move ahead in Rhode Island in a way we have not seen in any other state.

These are not easy changes, but they are necessary in order for the CCSS and other challenging reforms to be implemented successfully at scale.(EB)

Managing for Mastery

We have blogged about the topic of that last video post before, including a reference to Herzberg’s classic “One more time, what motivates employees?” And just like Herzberg, Daniel Pink points out that the three biggest factors that motivate people once the money is right is Autonomy (the desire to be self-directed), Mastery (the desire to get better at something), and Purpose (the desire to do something good). I ran across another article the other day about how they do human capital management at Google, and the same dynamic came through. Doing a good job seems to be the thing that we want. Companies that align their work and their purpose are flourishing. (Can you say, “Skype, Apple, and Whole Foods?”)

Given that our work is education, I am sure you can guess where this is all going. Race to the Top, the Gates Foundation, and a stalwart group of economists within the education reform sphere keep trying to incentivize high performing teachers (as measured by student growth) with bonus pay. We’ve talked about this before so we won’t belabor the point, but there is no evidence that pay motivates higher performance when you’re talking about complex work that requires thought, and if you’ve watched yesterday’s video, you now have another data point.

But what DOES seem to be motivating? Mastery, Autonomy, and Purpose. Education has at least one of these going for it right out of the gate: Purpose. And if you talk to teachers and principals like we do, you know that there is nothing more demotivating than having the “instructional coach” or “state observer” come into your classroom to watch your instruction for five minutes to tell you what you should be doing better. The autonomy variable is definitely at play here. To us, the trick in education, and with principals and teachers specifically, is how do we foster Mastery through our management?

Here is what we have seen: When student assessment data or classroom observation data is presented in a disaggregated way (vs. summarizations) and is turned around in a quick time frame after collecting the data (no more than one week), educators are much more likely to see the value of the data as a way to get better (or gain mastery). But when the turnaround of the same data is slow or the emphasis is on an aggregated “rating,” it becomes deeply demotivating, and in many cases fuels the political fire to slow down or stop the district or state’s reform efforts.

If purpose, mastery, and autonomy yield higher performance among teachers and principals, what does this then mean for the work of managers at the district level? And for the program designers at the state? We’d love to hear your opinion. (BR)

Motivation Animation

Every once and a while that friend that sends you three forwards a day hits on something interesting.  The other day, I received a link to a YouTube video from RSA that is a very entertaining visual walk through by Daniel Pink of the point we made on this blog about a year ago.  Enjoy! (BR)

Predicting Crime with CompStat?

A great article in Slate from Christopher Beam highlights a CompStat program in Los Angeles which will begin to use predictive statistics alongside traditional CompStat figures.  CompStat traditionally tracks a slate of common crime stats for each precinct commander every two weeks to help focus that commander on the results of their tactics over that period.  This data normally includes statistics on crime incidents like robberies, assaults, and homicides as well as crime related measures like complaints and arrests.  The idea is to diagnose why crime seems to have happened and to deploy police resources to mitigate those factors.

But as the article points out, the process looks backwards.  In Los Angeles and Santa Cruz, statisticians have crunched the numbers to learn that certain events predict the occurrence of crime with some regularity.  A home robbery ups the odds that a repeat robbery will happen in the area.  A gang shooting increases the odds of reprisal.  And as research continues, the LA Police are bound to find other predictors that Precinct Commanders can use to strategically deploy their forces and keep their communities safer.

Who knew Policing would take some cues from Education after all these years of CompStat inspiring SchoolStat? Since 2007, we’ve seen similar predictive work with the use of early warning indicators to predict the risk of student’s dropping out of high school.  Based on research from the Chicago Consortium of School Research, a high school student’s course performance is the single most predictive factor in whether a student will complete high school.  Specifically, CCSR concluded that Chicago students who finish ninth grade with at least ten semester credits or five full-year course credits and have no more than one semester F in a core course are nearly four times more likely to graduate than those who do not.  CCSR used this finding to create an On-Track to Graduate Indicator for current students who complete ninth grade with five credits in core courses and no more than one semester F.  Principals use this data to deploy counseling resources .

Rhode Island has an early warning indicator planned statewide for its Race to the Top program to address dropouts as well, and will prove a great addition to the EdStat process driving their RTT reforms.  Where else can you see predictive indicators taking hold? (BR)

CompStat and Campbell’s Law

As you may have seen in the news, The New York City Police Department is conducting a comprehensive review of its crime stats.  Over the past months, reports have emerged that Precinct Commanders felt pressured to downgrade serious crimes to less serious crimes to both look good at their CompStat sessions and ensure that the overall crime rates did not climb upward.

This case brings to mind an oft forgotten idea in public policy called Campell’s Law.  Cambell’s Law is an idea posited by American social scientist Donald Campbell stating that, “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”  As you can imagine, Campbell’s Law has been cited by many others in conversations about high stakes test scores, but it is important to remember that we see singular performance indicators driving bad behavior in just about ALL sectors.  Think no further than quarterly profit statements for Enron and WorldCom or loan sales at your favorite mortgage brokers (if they are still around).

So did CompStat and the drive to keep crime low in New York City “distort the social processes it was intended to monitor”?  I don’t think we’ll know the answer for a while, but as we’ve begun developing a statewide Stat process for the Race to the Top work in Rhode Island, we’ve been reminded of what the Stat process does in a new environment.  Whether it be CompStat in New York when it began under Bill Bratton or in any Stat process we develop with a client, the purpose is two fold.  First, it is to place the attainment of specific results at the forefront of a managers thinking as they make decisions about tactics, strategies, and resource deployment.  Second, it is to use the data itself in many disaggregated forms to inform and enrich the quality of our decisions and to objectively learn from past hypothesis on what works.  No one would argue that using this data in this way is bad management and “distorts the process it is intended to monitor”.  But at the end of the day, the use of data in management does not cure an organization of unsavory behavior, it simply changes the leverage points of where it can happen.

We’ve also been reminded of the importance of multiple measures.  Whether it be value added in teacher evaluation, test scores in AYP decisions for schools, or “crime” in CompStat, one measure never tells the whole story.  A good Stat process marries outcome metrics with survey, financial, and observational information to ensure that what gets measures not only gets done, but is what you want (BR)

Don’t Cross the Streams

A quick report from the trenches. We’ve been working these past weeks in Rhode Island helping stand up their more complex projects with Race to the Top and to help them performance manage the many complex streams of work that will happen in parallel.

Of note, we’ve been helping develop their educator evaluation program which will bring all teachers in the state onto a common evaluation platform that incorporates observations, goal attainment, and multiple measures of student growth. As the state has worked to include the maximum array of grades and subjects into the process, we’ve run into a challenge that I am sure many states will see as well.

If a state or district uses value-added in a teacher’s evaluation and the state test is the feed for the model, you limit the grades and teachers that the value-added model can cover to between 15 percent to 20 percent of teachers. That leaves a lot of teachers out of the program. In an effort to include more grades and subjects, many states are scrambling to find more assessments to inject into the model. In this search, some states and districts are considering the use of formative and interim assessments that track student progress against state standards or curriculum throughout the year.

There is a big problem with this. Formative data is held separate from summative data for very good reasons. Summative data is designed to tell you which students met academic standards for AYP designations. When students take the summative tests, teachers teach them “test taking strategies” to help them do the best they can on items where they are not completely sure.

The opposite is true of formative assessments. Teachers use formative assessments to understand the connection, or lack of connection, between what they are teaching, and what their students are learning. It is meant to be honest and accurate. If a student does not know the answer, the teacher tells them not to try and guess. The result is a more accurate picture of the specific areas of strength and weakness where the teacher can re-tool instruction.

So imagine for a second if these states and districts incorporate formative and interim data into a teacher’s evaluation. Yes, it might be a good picture of what a teacher’s students know, but you have just upended the purpose of that formative assessment and destroyed its value. If a teacher knows a formative or interim assessment will be part of their evaluation, they will tell their students to represent that they know things that they do not and the teacher will lose a powerful tool in helping them meet the very goals the summative test is trying to measure.  Rhode Island realized this early on.

Its a lose-lose proposition and states and districts looking to incorporate student data into non-tested grades and subjects should resist the temptation to cross the streams (BR).

Education Reform and Counter Insurgency

Our good friend Justin Cohen over at the “Turnaround Challenge” hit it spot on the in an entry on the relationship between good policy and good execution.  Justin mentions a Matt Yglesias quote on (of all the things to compare to education reform) counter insurgency strategy.  Yglesias says,

“… you can’t initiate a large complicated undertaking that involves coordinated action by hundreds of thousands of individual human beings and then make success contingent on perfect implementation.”

Fresh from a day of pondering state Race to the Top strategy, Justin notes, “I’m increasingly frustrated by the extent to which [education] policy discussions are execution-agnostic.”  We’ve been helping three states implement their Race to the Top, and we’ve seen the same thing from the front line.

Think about it.  An RTT winner has to now coordinate at least 20 separate new and inter-woven (not to mention politically risky) projects internally AND monitor and support the progress of around 10 projects at each of the school districts participating in their program (which could be as low as 55 and as high as more than 700 depending on the state).  This is a management super-lift in organizations that have rarely been rewarded for or capable of managing large complicated projects on their own.  Yet, when we look at any state’s application or at a district’s scope of work, we see work plans written as if they weren’t doing anything else, there was no angry teacher’s union waiting for them to mess up, and they have a bench of Harvard MBAs.  They are assuming near perfect implementation.

Our advice to these states has been to design themselves around the inevitability of imperfect implementation.  In education reform generally, and in RTT specifically, there is no recipe or checklist that we can follow for it to work.  We must instead live in a constant cycle of making a hypothesis of the best path forward, executing in earnest, reflecting frequently on our progress, mid-course correcting, repeat.

We’ll get into this in more detail in the weeks ahead. (BR)