Taking a Closer Look at Value Added

Last month I joined a team of UPD-ers and traveled around the state of Oklahoma training district-level trainers on Value-Added.  During one of the sessions, a participant raised his hand and asked our team how value added could be relied upon as a valid measure of teacher effectiveness when districts like Houston Independent School District[1] are currently involved in lawsuits surrounding the legitimacy of their value-added model, and the American Statistical Association (ASA) released a statement[2] that has been described as “slamming the high-stakes ‘value-added method’ (VAM) of evaluating teachers.”    Although we were familiar with both the Houston lawsuits and the ASA statement, this question created an opportunity to take a closer look at recent articles and information opposing (or seeming to oppose) value added.

 

First, a little background:  According to our partners at Mathematica Policy Research, “Value-added methods (sometimes described as student growth models) measure school and teacher effectiveness as the contribution of a school or teacher to students’ academic growth. The methods account for students’ prior achievement levels and other background characteristics.”  Value added does this via a statistical model that is based on educational data from the given state or district, and uses standardized test scores to evaluate teachers’ contribution to student achievement. Although value added and similar measures of student growth had been used in various places in the United States without much opposition, criticism peaked around 2010 when districts such as Chicago, New York City and Washington, DC began incorporating value-added into high-stakes teacher evaluation models.  Since then various individuals and organizations have published their views on the merits or pitfalls of value added including, most recently, the American Statistical Association (ASA).

 

The ASA statement has garnered considerable attention because as described by Sean McComb, 2014 National Teacher of the Year, “… I thought that they are experts in statistics far more than I am. So I thought there was some wisdom in their perspective on the matter.”[3] Of course as statistical experts they shed some light on what can and cannot reasonably be expected from the use of value-added measures, but here are a few ways that we can address parts of their statement that may be misunderstood:

• The ASA mentions that value added models “are complex statistical models, and high-level statistical expertise is needed to develop the models and interpret their results. Estimates from VAMs should always be accompanied by measures of precision and a discussion of the assumptions and possible limitations of the model.”  Although it is true that the models themselves are complex and require advanced statistical expertise to compute, we would argue that people without this level of expertise can be trained on the concepts behind how the models work and also how results should be interpreted.  In Oklahoma, part of the training we provide is designed to help teachers build a conceptual understanding of the statistics behind value added.  Although we do not look at the regression formula itself, we help to define components of the measure including how it is developed, its precision, etc. so that teachers are able to better understand how value added can provide additional data to help inform their instruction.
• In the report, the ASA cautions that since value added is based on standardized test scores, and other student outcomes are predicted only to the extent that they correlate with test scores, it does not adequately capture all aspects of a teachers effectiveness – “A teacher’s efforts to encourage students’ creativity or help colleagues improve their instruction, for example, are not explicitly recognized in VAMs.”  This statement is true and it is one that we are quick to highlight when we train on value added.  Value-added models are not designed to measure teacher effectiveness in isolation as they only tell part of the story.  When used as part of an evaluation system with multiple measures (such as classroom observations and student surveys), a more complete and stable picture becomes available.
• Finally the ASA clearly states that “VAM scores are calculated using a statistical model, and all estimates have standard errors. VAM scores should always be reported with associated measures of their precision, as well as discussion of possible sources of biases.”[4] Since we are always transparent about the fact that all value-added estimates have confidences intervals, this is almost always something that trips people up during training sessions.  Many will say, “If there is a margin of error, then how can this measure be trusted enough to include in an educator evaluation system?”   What is easy to forget is that all measures, statistical or not, come with some level of uncertainty.  This includes more traditional methods of teacher evaluation such as classroom observations.  Although efforts should be made to limit or decrease the margin of error where possible, there will never be a way to completely eliminate all error from something as wide and deep as teacher effectiveness. Despite this, this does not mean that value added should not be used to evaluate teachers but, as mentioned previously, it should be considered alongside other measures.

 

By Titilola Williams-Davies., a consultant at UPD Consulting.

 

 

 

[1]Strauss, Valerie. April 30, 2014. ”Houston teachers’ lawsuit against the Houston Independent School District” Washington Post. http://apps.washingtonpost.com/g/documents/local/houston-teachers-lawsuit-against-the-houston-independent-school-district/967/

 

[2]American Statistical Association. April 8, 2014. “ASA Statement on Using Value-Added Models for Educational Assessment.” http://www.amstat.org/policy/pdfs/ASA_VAM_Statement.pdf

 

[3] Valerie Strauss. April 30, 2014. “2014 National Teacher of the Year: Let’s stop scapegoating teachers” Washington Post. http://www.washingtonpost.com/blogs/answer-sheet/wp/2014/04/30/2014-national-teacher-of-the-year-lets-stop-scapegoating-teachers/?tid=up_next

 

[4] American Statistical Association. April 8, 2014. “ASA Statement on Using Value-Added Models for Educational Assessment.” http://www.amstat.org/policy/pdfs/ASA_VAM_Statement.pdf

 

Teacher Evaluation and the Burden of Evidence

Over the last several years, school districts across the country have been rolling out more rigorous and detailed evaluation rubrics along with systems to help administrators and teachers collaborate, collect evidence, and share performance ratings. If your school district (or school district that you are consulting for) is adopting a new teacher evaluation system to aid with collecting quality evidence, I have a single piece of advice for you. But you have to read to the end to get it.

The new evaluation frameworks are significantly more complex than the evaluation practices they have replaced. The Danielson framework, for example, contains 22 potential components. These are containers or categories for what is commonly referred to as “evidence,” typically taken in the form of notes by administrators. In some school districts as many as 10 of these 22 components need to be captured during a single class period teacher observation. At the end of the day, in districts that have already rolled out new rubrics, the burden of evidence to support those rubrics falls heavily on the administrators.

Now, I want to magically transport you and your laptop to the back of a classroom where it is your job to capture all applicable evidence for a complicated new framework, that has just been introduced to you, while the teacher works through the ebbs and flows of a lesson and the students fire off questions. Ready, set, go! The teacher begins the lesson and you have become the equivalent of a court reporter. How good are your shorthand typing skills? How dependable is your wi-fi? Can you really type up quality evidence and organize it on the spot, using new technology?

 

 Photo by Julia Kuo

From my experience having personally trained hundreds of Principals and Assistant Principals on teacher evaluation tools, this may be the single most valuable lesson I’ve learned: How can you change years, or even decades, of note-taking habits over the course of a few hours and show administrators how to collect evidence in an entirely different way? Easy answer: you don’t.

If you are going to roll out detailed rubrics and build sophisticated systems to capture the evidence, that is just swell. But when it comes to actually capturing the notes in the classroom session, I urge you to empower the administrators to use their preferred methods of colleting notes, whatever those methods may be. Spreadsheet? Sure. Word processing? Of course. Pen and paper? Knock yourself out (though you will have to type it up later). The important thing is that the administrator is comfortable enough to capture notes quickly and this ensures that they aren’t missing valuable evidence while they try to tango with your new system or their potentially lousy wi-fi connection. The frameworks have likely changed, so I don’t mean to imply that the quality and content of notes captured by the administrator don’t have to change: they do. What I am proposing is that you don’t handicap them with a single tool in the fast-paced environment of classroom observation.

You can and should design a system this way. We have done such a thing with Truenorthlogic, who has rolled out their software to some of the largest school districts in the country. Evidence can be copied and pasted from multiple sources. Splitting up the evidence and grouping it by the rubric is incredibly flexible and doesn’t have to be done on the spot in the classroom.

When we piloted the teacher evaluation system at the Chicago Public Schools, we first proposed that administrators replace their previous evidence collection entirely and fire up their laptops and tablets to collect evidence directly in the new system. This did not go over nearly as well as our adjusted approach in the district-wide rollout, where we adopted the “take notes however you like” approach. By then, we had learned that administrators need to be empowered to collect evidence using their preferred method. We need their brain power focused primarily on capturing quality, relevant evidence to their new framework, not on navigating new technology. As long as they can turn around and input that evidence in the system efficiently and share it with the teacher by the required deadline, more power to them. Everyone wins.

 

 

By Frank Nichols. Frank Nichols is a Consultant at UPD Consulting.