The Baltimore Consensus

In 2008, the Copenhagen Consensus Center asked a group of the world’s top economists to identify optimal social “investments” that could best help reduce malnutrition, broaden educational opportunity, slow global warming, cut air pollution, prevent conflict, fight disease, improve access to water and sanitation, lower trade and immigration barriers, thwart terrorism, and promote gender equality.

The experts—including five Nobel laureates—examined specific measures to spend $75 billion on more than 30 interventions and indentified the most cost-effective: increased immunization coverage, initiatives to reduce school dropout rates, community-based nutrition promotion, and micronutrient supplementation.  Besides being resource efficient, some of these measures are also very low cost-per-user, such as micronutrient supplementation: providing Vitamin A for a year costs as little as $1.20 per child, while providing Zinc costs as little as $1.

This got us at UPD thinking: what would a Copenhagen Consensus in American K-12 look like?  After all, in an age of severe budget pressures, we need to know the best measures that boards and superintendents can implement to help boost student performance.  And it would be great if those high impact measures were low cost, so we pushed ourselves to find ideas that would not require vast new resources.

Our top nine ideas share two themes: leveraging existing data and technology investments to improve instruction and enhancing human capital management.  None of our suggestions require new expenses, though they will require changes in culture and time use. Here’s our top nine:

1. Routinely examine data that comes from formative assessment data with groups of teachers, principals, and curriculum and instruction managers.  Provide the data ahead of time.
2. Implement human capital reforms that bring mutual consent to all teacher hiring.
3. Integrate student results into the performance evaluations of teachers.
4. Establish performance management/accountability processes at all levels of the organization, from central office functions to RTI in classrooms.
5. Improve targeting of professional development needs and resources in order to make average teachers better.
6. Decentralize dollars and control to the school level coupled with changes in how principals are hired and evaluated (more like coaches in professional sports).
7. Systematically capture data on student, teacher, principal participation in different interventions to effectively discern contributors to high performance.
8. Leverage technology to automatically provide parents and guardians with content that helps them supplement the scope and pacing of student curriculum.
9. Use predictive analytics to uncover students with likely future behavioral difficulties very early and mount high-impact interventions before its too difficult (JG).

What are your picks?

Predicting Crime with CompStat?

A great article in Slate from Christopher Beam highlights a CompStat program in Los Angeles which will begin to use predictive statistics alongside traditional CompStat figures.  CompStat traditionally tracks a slate of common crime stats for each precinct commander every two weeks to help focus that commander on the results of their tactics over that period.  This data normally includes statistics on crime incidents like robberies, assaults, and homicides as well as crime related measures like complaints and arrests.  The idea is to diagnose why crime seems to have happened and to deploy police resources to mitigate those factors.

But as the article points out, the process looks backwards.  In Los Angeles and Santa Cruz, statisticians have crunched the numbers to learn that certain events predict the occurrence of crime with some regularity.  A home robbery ups the odds that a repeat robbery will happen in the area.  A gang shooting increases the odds of reprisal.  And as research continues, the LA Police are bound to find other predictors that Precinct Commanders can use to strategically deploy their forces and keep their communities safer.

Who knew Policing would take some cues from Education after all these years of CompStat inspiring SchoolStat? Since 2007, we’ve seen similar predictive work with the use of early warning indicators to predict the risk of student’s dropping out of high school.  Based on research from the Chicago Consortium of School Research, a high school student’s course performance is the single most predictive factor in whether a student will complete high school.  Specifically, CCSR concluded that Chicago students who finish ninth grade with at least ten semester credits or five full-year course credits and have no more than one semester F in a core course are nearly four times more likely to graduate than those who do not.  CCSR used this finding to create an On-Track to Graduate Indicator for current students who complete ninth grade with five credits in core courses and no more than one semester F.  Principals use this data to deploy counseling resources .

Rhode Island has an early warning indicator planned statewide for its Race to the Top program to address dropouts as well, and will prove a great addition to the EdStat process driving their RTT reforms.  Where else can you see predictive indicators taking hold? (BR)

Why We Still Can’t Understand Value-Added

Analogies are important. While often imperfect, they help us make connections and better understand our world.

That’s why I think it’s a shame the education field has yet to come up with good analogies for value-add data and models, especially in the teacher evaluation context. For example:

• Is value-add like a high-school GPA, providing a threshold for certain decisions (you can’t get into College ABC without a certain GPA) but best used in conjunction with other factors (were you a varsity athlete? Do you do community service? How was your admissions essay?)
• Is value-add like a batting average, telling us how good teachers are at some skills (hitting), but not others (fielding)?
• Is value-add like a credit score, highly dependent on input qualities (did the bank get everything right?) with the potential to change over time (decreasing when you took on student loans, increasing as your credit history grows)?

The inability of education reformers to clearly explain value-add has confused the conversation. For example, yesterday I went to a terrific conference sponsored by the National Center for the Analysis of Longitudinal Data in Education Research (a mouthful! – so they go by CALDER). The brightest minds in education research presented papers on value-add. But it was clear that throughout the day, the non-researchers in the audience struggled to grasp the policy implications. In a somewhat tense moment (as these things go), an audience member suggested that value-add doesn’t take into consideration x situation so it can’t be used by itself to assess teachers, to which an exasperated panel member replied “I just heard [my colleague] say that!” Something was clearly lost in translation – an analogy would have been useful (JF).

What analogies would you offer for value-add?

CompStat and Campbell’s Law

As you may have seen in the news, The New York City Police Department is conducting a comprehensive review of its crime stats.  Over the past months, reports have emerged that Precinct Commanders felt pressured to downgrade serious crimes to less serious crimes to both look good at their CompStat sessions and ensure that the overall crime rates did not climb upward.

This case brings to mind an oft forgotten idea in public policy called Campell’s Law.  Cambell’s Law is an idea posited by American social scientist Donald Campbell stating that, “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”  As you can imagine, Campbell’s Law has been cited by many others in conversations about high stakes test scores, but it is important to remember that we see singular performance indicators driving bad behavior in just about ALL sectors.  Think no further than quarterly profit statements for Enron and WorldCom or loan sales at your favorite mortgage brokers (if they are still around).

So did CompStat and the drive to keep crime low in New York City “distort the social processes it was intended to monitor”?  I don’t think we’ll know the answer for a while, but as we’ve begun developing a statewide Stat process for the Race to the Top work in Rhode Island, we’ve been reminded of what the Stat process does in a new environment.  Whether it be CompStat in New York when it began under Bill Bratton or in any Stat process we develop with a client, the purpose is two fold.  First, it is to place the attainment of specific results at the forefront of a managers thinking as they make decisions about tactics, strategies, and resource deployment.  Second, it is to use the data itself in many disaggregated forms to inform and enrich the quality of our decisions and to objectively learn from past hypothesis on what works.  No one would argue that using this data in this way is bad management and “distorts the process it is intended to monitor”.  But at the end of the day, the use of data in management does not cure an organization of unsavory behavior, it simply changes the leverage points of where it can happen.

We’ve also been reminded of the importance of multiple measures.  Whether it be value added in teacher evaluation, test scores in AYP decisions for schools, or “crime” in CompStat, one measure never tells the whole story.  A good Stat process marries outcome metrics with survey, financial, and observational information to ensure that what gets measures not only gets done, but is what you want (BR)

UPD’s Otis Rolley at TedX MidAtlantic

Check out Otis Rolley, Senior Manager for UPD’s Community Development practice, giving a thought-provoking talk at a recent TedX event in Washington.

CitiStat and Law and Order

Did you know that the Law and Order guy, Sam Waterson, did a video on CitiStat?  Check out this video with real shots of meetings and interviews with the originators, then Mayor Martin O’ Malley, Michael Enright, and Matt Gallagher.

Don’t Cross the Streams

A quick report from the trenches. We’ve been working these past weeks in Rhode Island helping stand up their more complex projects with Race to the Top and to help them performance manage the many complex streams of work that will happen in parallel.

Of note, we’ve been helping develop their educator evaluation program which will bring all teachers in the state onto a common evaluation platform that incorporates observations, goal attainment, and multiple measures of student growth. As the state has worked to include the maximum array of grades and subjects into the process, we’ve run into a challenge that I am sure many states will see as well.

If a state or district uses value-added in a teacher’s evaluation and the state test is the feed for the model, you limit the grades and teachers that the value-added model can cover to between 15 percent to 20 percent of teachers. That leaves a lot of teachers out of the program. In an effort to include more grades and subjects, many states are scrambling to find more assessments to inject into the model. In this search, some states and districts are considering the use of formative and interim assessments that track student progress against state standards or curriculum throughout the year.

There is a big problem with this. Formative data is held separate from summative data for very good reasons. Summative data is designed to tell you which students met academic standards for AYP designations. When students take the summative tests, teachers teach them “test taking strategies” to help them do the best they can on items where they are not completely sure.

The opposite is true of formative assessments. Teachers use formative assessments to understand the connection, or lack of connection, between what they are teaching, and what their students are learning. It is meant to be honest and accurate. If a student does not know the answer, the teacher tells them not to try and guess. The result is a more accurate picture of the specific areas of strength and weakness where the teacher can re-tool instruction.

So imagine for a second if these states and districts incorporate formative and interim data into a teacher’s evaluation. Yes, it might be a good picture of what a teacher’s students know, but you have just upended the purpose of that formative assessment and destroyed its value. If a teacher knows a formative or interim assessment will be part of their evaluation, they will tell their students to represent that they know things that they do not and the teacher will lose a powerful tool in helping them meet the very goals the summative test is trying to measure.  Rhode Island realized this early on.

Its a lose-lose proposition and states and districts looking to incorporate student data into non-tested grades and subjects should resist the temptation to cross the streams (BR).