Taking a Closer Look at Value Added

Last month I joined a team of UPD-ers and traveled around the state of Oklahoma training district-level trainers on Value-Added.  During one of the sessions, a participant raised his hand and asked our team how value added could be relied upon as a valid measure of teacher effectiveness when districts like Houston Independent School District[1] are currently involved in lawsuits surrounding the legitimacy of their value-added model, and the American Statistical Association (ASA) released a statement[2] that has been described as “slamming the high-stakes ‘value-added method’ (VAM) of evaluating teachers.”    Although we were familiar with both the Houston lawsuits and the ASA statement, this question created an opportunity to take a closer look at recent articles and information opposing (or seeming to oppose) value added.

 

First, a little background:  According to our partners at Mathematica Policy Research, “Value-added methods (sometimes described as student growth models) measure school and teacher effectiveness as the contribution of a school or teacher to students’ academic growth. The methods account for students’ prior achievement levels and other background characteristics.”  Value added does this via a statistical model that is based on educational data from the given state or district, and uses standardized test scores to evaluate teachers’ contribution to student achievement. Although value added and similar measures of student growth had been used in various places in the United States without much opposition, criticism peaked around 2010 when districts such as Chicago, New York City and Washington, DC began incorporating value-added into high-stakes teacher evaluation models.  Since then various individuals and organizations have published their views on the merits or pitfalls of value added including, most recently, the American Statistical Association (ASA).

 

The ASA statement has garnered considerable attention because as described by Sean McComb, 2014 National Teacher of the Year, “… I thought that they are experts in statistics far more than I am. So I thought there was some wisdom in their perspective on the matter.”[3] Of course as statistical experts they shed some light on what can and cannot reasonably be expected from the use of value-added measures, but here are a few ways that we can address parts of their statement that may be misunderstood:

• The ASA mentions that value added models “are complex statistical models, and high-level statistical expertise is needed to develop the models and interpret their results. Estimates from VAMs should always be accompanied by measures of precision and a discussion of the assumptions and possible limitations of the model.”  Although it is true that the models themselves are complex and require advanced statistical expertise to compute, we would argue that people without this level of expertise can be trained on the concepts behind how the models work and also how results should be interpreted.  In Oklahoma, part of the training we provide is designed to help teachers build a conceptual understanding of the statistics behind value added.  Although we do not look at the regression formula itself, we help to define components of the measure including how it is developed, its precision, etc. so that teachers are able to better understand how value added can provide additional data to help inform their instruction.
• In the report, the ASA cautions that since value added is based on standardized test scores, and other student outcomes are predicted only to the extent that they correlate with test scores, it does not adequately capture all aspects of a teachers effectiveness – “A teacher’s efforts to encourage students’ creativity or help colleagues improve their instruction, for example, are not explicitly recognized in VAMs.”  This statement is true and it is one that we are quick to highlight when we train on value added.  Value-added models are not designed to measure teacher effectiveness in isolation as they only tell part of the story.  When used as part of an evaluation system with multiple measures (such as classroom observations and student surveys), a more complete and stable picture becomes available.
• Finally the ASA clearly states that “VAM scores are calculated using a statistical model, and all estimates have standard errors. VAM scores should always be reported with associated measures of their precision, as well as discussion of possible sources of biases.”[4] Since we are always transparent about the fact that all value-added estimates have confidences intervals, this is almost always something that trips people up during training sessions.  Many will say, “If there is a margin of error, then how can this measure be trusted enough to include in an educator evaluation system?”   What is easy to forget is that all measures, statistical or not, come with some level of uncertainty.  This includes more traditional methods of teacher evaluation such as classroom observations.  Although efforts should be made to limit or decrease the margin of error where possible, there will never be a way to completely eliminate all error from something as wide and deep as teacher effectiveness. Despite this, this does not mean that value added should not be used to evaluate teachers but, as mentioned previously, it should be considered alongside other measures.

 

By Titilola Williams-Davies., a consultant at UPD Consulting.

 

 

 

[1]Strauss, Valerie. April 30, 2014. ”Houston teachers’ lawsuit against the Houston Independent School District” Washington Post. http://apps.washingtonpost.com/g/documents/local/houston-teachers-lawsuit-against-the-houston-independent-school-district/967/

 

[2]American Statistical Association. April 8, 2014. “ASA Statement on Using Value-Added Models for Educational Assessment.” http://www.amstat.org/policy/pdfs/ASA_VAM_Statement.pdf

 

[3] Valerie Strauss. April 30, 2014. “2014 National Teacher of the Year: Let’s stop scapegoating teachers” Washington Post. http://www.washingtonpost.com/blogs/answer-sheet/wp/2014/04/30/2014-national-teacher-of-the-year-lets-stop-scapegoating-teachers/?tid=up_next

 

[4] American Statistical Association. April 8, 2014. “ASA Statement on Using Value-Added Models for Educational Assessment.” http://www.amstat.org/policy/pdfs/ASA_VAM_Statement.pdf

 

Managing for Mastery

We have blogged about the topic of that last video post before, including a reference to Herzberg’s classic “One more time, what motivates employees?” And just like Herzberg, Daniel Pink points out that the three biggest factors that motivate people once the money is right is Autonomy (the desire to be self-directed), Mastery (the desire to get better at something), and Purpose (the desire to do something good). I ran across another article the other day about how they do human capital management at Google, and the same dynamic came through. Doing a good job seems to be the thing that we want. Companies that align their work and their purpose are flourishing. (Can you say, “Skype, Apple, and Whole Foods?”)

Given that our work is education, I am sure you can guess where this is all going. Race to the Top, the Gates Foundation, and a stalwart group of economists within the education reform sphere keep trying to incentivize high performing teachers (as measured by student growth) with bonus pay. We’ve talked about this before so we won’t belabor the point, but there is no evidence that pay motivates higher performance when you’re talking about complex work that requires thought, and if you’ve watched yesterday’s video, you now have another data point.

But what DOES seem to be motivating? Mastery, Autonomy, and Purpose. Education has at least one of these going for it right out of the gate: Purpose. And if you talk to teachers and principals like we do, you know that there is nothing more demotivating than having the “instructional coach” or “state observer” come into your classroom to watch your instruction for five minutes to tell you what you should be doing better. The autonomy variable is definitely at play here. To us, the trick in education, and with principals and teachers specifically, is how do we foster Mastery through our management?

Here is what we have seen: When student assessment data or classroom observation data is presented in a disaggregated way (vs. summarizations) and is turned around in a quick time frame after collecting the data (no more than one week), educators are much more likely to see the value of the data as a way to get better (or gain mastery). But when the turnaround of the same data is slow or the emphasis is on an aggregated “rating,” it becomes deeply demotivating, and in many cases fuels the political fire to slow down or stop the district or state’s reform efforts.

If purpose, mastery, and autonomy yield higher performance among teachers and principals, what does this then mean for the work of managers at the district level? And for the program designers at the state? We’d love to hear your opinion. (BR)

Motivation Animation

Every once and a while that friend that sends you three forwards a day hits on something interesting.  The other day, I received a link to a YouTube video from RSA that is a very entertaining visual walk through by Daniel Pink of the point we made on this blog about a year ago.  Enjoy! (BR)

Value-Added Data and Special Education

At a gala for the American Association of People with Disabilities in March, Education Secretary Arne Duncan affirmed the current administration’s commitment to maintaining high expectations for special education populations, noting that “students with disabilities should be judged with the same accountability system as everyone else.” While most educators would readily support this goal, they would also probably tell you that achieving it is a lot easier said than done—especially when it comes to using student achievement data as a factor in evaluating special education teachers.

In an education reform landscape that seems saturated with increasingly complex questions about accountability systems (particularly around the use of value-added models in educator evaluation), determining where special education students and teachers fit into those systems poses some of the most complex questions of all. So what progress have we made to determine how value-added data should be used to measure achievement in special education students? The answer seems to be…not that that much.

There are plenty of pretty obvious reasons why value-added models pose fundamental problems in the special education world. One potentially insurmountable obstacle is the lack of standardized test scores. Most value-added models require at least two years’ worth of test data for each student. This makes it nearly impossible to collect value-added data for students with severe cognitive disabilities that qualify for their state’s alternate assessment. Alternative assessments, which were mandated as part of the reauthorization of IDEA in 1997, are scored on completely different scales than the state standardized tests. While some states have attempted to scale the scores and create comparable data for completing value-added analysis, most have chosen to exclude this group of students completely.

Assessment experts have also pointed out that the results that alternative assessments yield lack the “fine tuning” that is needed to complete value-added calculations with confidence. Although there is a strong push by the US Department of Education to substantially reduce the number of students with disabilities taking the alternate assessment (which is expected to be backed by the reauthorization of the Elementary and Secondary Education Act coming next fall), it will be years before states even have the option of including students from this group as part of their value-added calculations.

The challenges aren’t limited to using value-added data to measure progress for special education students who are taking the alternate assessment. A report by the National Comprehensive Center for Teacher Quality issued last July identified a number of obstacles that impact a wider group of students, including the fact that researchers have yet to identify an appropriate way to account for the impact of testing accommodations on test scores of special education students who take the regular state test.

Without a way to control for the impact of testing accommodations on student performance, the testing data from this group of students is difficult (if not impossible) to use to draw precise conclusions about the “value” added by special education teachers. Although states continue to work tirelessly to develop educator evaluation systems that incorporate value-added data, efforts to find new ways to incorporate precise measures that capture student achievement in the context of special educators’ evaluations seem to be lagging behind. While the challenges listed above (among a host of others) may represent valid reasons why standard value-added models may not work with special education data, there is important work to be done in developing other means for determining precise measures of progress for special education students.

This is not to say that special education teachers are excluded from the emerging high stakes evaluation models—they certainly aren’t. States have developed a variety of alternatives to using value-added data for evaluating special education teachers, but the accuracy and precision of the information they provide has far less backing by research than the models applied to general education populations. If the measures used to determine the effectiveness of special education teachers aren’t as precise as those used for general education teachers, states and districts will be limited in their ability use that data to drive meaningful professional development and support.

In a field that is historically lacking in quality professional development, it seems that states are missing a valuable opportunity to use their evaluation systems to make vast improvements in the quality of support special educators are afforded. If we aren’t doing enough to determine how to measure progress accurately for special education students, it means that we aren’t doing enough to support special education teachers in becoming more effective. (JS)