inBloom, Train Wrecks, and Ed-Fi

As I sat down to write this entry, my day was interrupted most unusually.  Doug texted me the picture to the left.  The caption said simply, “Say hello to 26th street and the railroad track.”  In the picture I saw the same view I see every work morning from the “Big Table” here at UPD where many of us sit.  After more than 4 inches of rain over 36 hours, the ground right outside our office gave way taking more than a dozen cars and half the street with it.  If you watch the video (found below) of the ground as it collapses underneath the cars, you will see that it left the wall with nothing to hold, and fell under its own weight.  The stories on the news have since revealed that the neighborhood has know this was a problem for years, but their complaints and concerns met a deaf ear in the city and with the rail company.

 

It’s hard to see such a calamity and think not metaphorically about my originally intended subject: the collapse of inBloom. inBloom was, in lieu of a more boring technical description, a cloud based data integration technology, that would enable districts and states to connect their data to an “app store” of programs and dashboards that could sit on top.  The vision was a seamless and less expensive way for teachers and principals to gain easy access to data about their students.

 

inBloom was a very big deal.  Started in 2011, several big funders and education heavies devoted their credibility and more than $100 million to try to make it successful.  Their efforts succeeded in garnering several state and district partners.  But since its inception, consumer groups, parents, and privacy advocates have worried that placing their students data in the hands of a third party would not be safe.  Or worse, inBloom might “sell” their student’s data to the highest bidder.  Then came Edward Snowden, and what was a niche news story went prime time.

 

If you look at the technology within inBloom that transfers and stores data in the cloud, the critics did not have much of a leg to stand on. inBloom’s data protection technology is as good or better than just about any existing state or district.  If you look at inBloom’s license agreement, parents and privacy advocates had more explicit protections than they have now with many student data systems.  What caused inBloom to collapse as quickly as the wall outside my window was more fundamental: trust.  As citizens, we trust districts and states with our students’ data.  And for all of inBloom’s technical explanations on the security of the data, they never made the case that we could trust them as an organization.  With the withdrawal of Louisiana, New York, Colorado, and several districts, nothing could hold inBloom up.

 

Over the past year at UPD, we’ve done a lot of work with the Ed-Fi data integration and dashboard suite.  We successfully rolled out the system for the entire State of South Carolina in about nine months (public dashboards here) and are very excited to start work with the Cleveland Metropolitan Public Schools to implement Ed-fi there.  Ed-Fi is very different than inBloom, even though they both utilize the same underlying data model.  Based on extensive research on what teachers and principals say they need, Ed-Fi provides a set of powerful data integration and dashboard tools that a district or state can download for free.  Rather than shooting data up into the cloud, Ed-Fi lives where most people already trust, in the data centers of the district or states.  19 states and more than 7,000 districts have licensed Ed-Fi.

 

The tragedy of inBloom is that it was a great idea ahead of its time and stood to do a lot of good in education.  But the protectors of the status quo should see no victory in its collapse.  Teachers and principals are clamoring for better information to help their students.  Ed-Fi seems ready to pick up where inBloom left off, and do so with the trust this work requires.

———————————

This blog was written by Bryan Richardson. Bryan is a Partner at UPD Consulting and brings over thirteen years of experience in private and public sector management. Bryan holds national expertise in performance management, data systems, and complex project implementation. 

Asking the Right Questions—Data Privacy and Security

There are a lot of good signs to be seen in recent news about security and privacy in the education technology sector. Some of the key questions being asked by educators and administrators are “how well are student data protected from prying eyes and greedy corporations?” and “who has access and how are the data being used?” These are good questions, and they represent the vestiges of our struggles with adopting modern technology over the past 15 years. Conversations have matured from simple arguments around the value of computers in every classroom to philosophical debates about our organizations’ embrace of performance data as the ombudsman of quality education. Progress has clearly been made, but in our rush to catch-up with our corporate cousins we missed asking what turns out to be a pretty important question–who owns all this stuff?

 

That’s the question that ultimately sealed the fate of inBloom, a non-profit offering a cloud-based data warehouse designed to help districts and vendors share student information. Despite funding from big foundation names like Gates and Carnegie, inBloom collapsed under the weight of a five word question they were never able to answer well enough to satisfy concerned stakeholders. If data are stored on a machine that is not physically located in a building owned by the district, who really owns the data?

The data issue is really a matter of security and access, which isn’t so different from the days of paper records in filing cabinets–information was kept in a secure, locked location and only certain people had access. With data warehouses replacing filing cabinets, the difference is that the information is stored off-site and the keys are also in the hands of the data warehouse manager (in other words, the system or database administrator). inBloom failed to effectively communicate this subtle difference early on, and any answer they eventually provided came across as reactionary, slick, dishonest, and–my favorite new term–“hand-wavy.”

 

Schools and districts aren’t used to asking those questions, and the education technology sector isn’t used to answering them. This disconnect doomed the effort from the beginning. Had the question “who owns this stuff?” been asked early on, the answer would have at least brokered a conversation rather than distrust and eventual dismissal–not to mention a waste of about $100 million dollars in grant funding.

 

Ideally, that conversation would lead to a compromise where information storage and archiving solutions satisfy the security and access needs of all players–parents, teachers, administrators, and the general public. Perhaps the right solution keeps an element of the status quo: secure data such as individual names, contact information, and other personally identifiable information could be stored on-site with the keys in the hands of the same people, but the bulk of the data could be stored in the cloud. Hybrid solutions like this are possible with dashboard software like Ed-Fi where the software itself can be installed on-site along with the secure data and set up to pull the remainder of the data from the cloud.

 

In the consulting world at UPD, we see those disconnect problems all the time: Group A spends a ton of time and money solving a problem for Group B without ever truly engaging the members of Group B. inBloom undoubtedly engaged their stakeholders in the early stages, but not deep enough to where someone was able to ask “who owns all this stuff?” This is often the result of too much focus on delivering a solution and providing answers rather than asking questions and identifying the problem. It comes with the territory–we get so excited about the possibilities of new technology that we jump right into requirements gathering without stopping to think if we’re asking the right questions and solving the right problem. It might just be as simple as an issue of maturity; if we’re getting serious about our relationship with technology, it’s probably time we start asking about intentions.

———————————

This blog was written by Andrew Keller. Andrew is a Consultant at UPD Consulting and brings over 10 years of experience in education, policy, and data metrics.  

Can Early Teacher Evaluation Findings Help Change the Debate?

Over the past few years, states and school districts across the country have devoted significant resources to the design and roll-out of new teacher evaluation systems.  Driven at least in part by requirements attached to Race to the Top funding, the new systems have inspired heated debate over the efficacy of factoring student achievement data into a teacher’s performance assessment. The New York Times recently shared some initial findings from states that have launched new evaluation models including Michigan, Florida and Tennessee, reporting that the vast majority of teachers- upwards of 95 percent in all three- were rated as effective or highly effective. Although the analysis of these numbers has only just begun, the Times reports that some proponents of the new evaluation models admit that the early findings are “worrisome”.  And even though it is still early, we can reasonably anticipate that if the trend continues- and the findings from the new evaluation systems reveal no significant departure from more traditional methods of evaluation- we may start to have a lot more people looking at the complicated data analysis driving teacher evaluation systems linked to student achievement data and asking “what’s the point?”

It’s a good question, really, and one that probably hasn’t gotten enough thoughtful attention in the midst of the controversy surrounding them: What is the point of linking student achievement data to teacher evaluations?  Should we take it for granted that a primary goal- if not the primary goal- of these efforts is to identify and eliminate bad teachers?  If this is the case then these early findings should be a cause for concern, especially given the time and money being spent to collect and analyze the data.  If replacing bad teachers with good ones is the magic bullet for public education reform, it will take a pretty long time at this rate.

Of course, even opponents of the new evaluation systems would probably admit that the magic bullet theory is an oversimplification. Furthermore, it’s much too early to look at these numbers and extrapolate any meaningful conclusions about the actual number of ineffective teachers or even the accuracy of the evaluations themselves. Hopefully what these findings might do is allow us to finally begin to broaden the scope of our national conversation about how the linkages between teachers and students could actually drive education reform.  States and school districts implementing new evaluation systems have tried with varying degrees of success to communicate the message that linking student achievement data to teacher practice isn’t just about punitive measures- it also has important implications for improving professional development and teacher preparation programs by identifying shared practice linked to positive student achievement and replicating those practices in classrooms across the country. But that message is often overshadowed by the anxiety surrounding the punitive side of evaluation and underscored by public struggles with local teacher unions. If nothing else, these early findings might create an opening in the current debate for a more thoughtful discussion about the broader possibilities for linking teacher practice to student growth.

-Jacqueline Skapik

Across the Board: Who Else is Responsible for the Atlanta Cheating Scandal?

The media in recent weeks has focused a great deal of attention on the cheating scandal in Atlanta in which authorities have indicted 35 officials and teachers in the Atlanta Public Schools system for allegedly altering the results of students’ state standardized tests to reflect higher scores.

There have been similar allegations in other schools and districts around the country.  What is unique about Atlanta, however, is the scope of the alleged fraud.  Those who work in education know that it is virtually impossible to keep a secret about even the most trivial matters in a large urban school district.  One cannot help but ask, therefore, how a scheme of this magnitude (with so many teachers and administrators directly involved) could gain such momentum.

By all accounts, it appears that pervasive in the culture of the district was the “Machiavellian” notion that the ends of increasing student test scores outweighed the means, and that it was this culture that enabled such widespread cheating.  Indeed, many point to an array of incentives that the superintendent allegedly used to reward those who generated higher test scores, whatever the cost to students.

The superintendent, however, does not operate in a vacuum.  Even the superintendent is accountable to the school board.  The superintendent alone cannot establish policy.  Nor can she create financial incentives for employees without the approval of the school board. She takes her cues as the tone of her leadership and school district culture from the school board.  If not, the school board should remove her.  Why then has there been so little media attention on the role of the board in the Atlanta cheating scandal?

The school board, at least as much as the superintendent, is responsible for creating a culture of integrity, honor, and accountability within the district.  Effective school boards model these values both in the conduct of school board meetings and in their interactions with school communities.

Moreover, effective school boards recognize and constantly communicate to others the critical importance of accurate data in improving instruction and learning outcomes.  They establish policies to ensure the integrity of test results.[i]  They examine student data on a routine basis, and hold district and school administrators accountable for the effective use of the data.  In so doing, the board ought to catch extreme abnormalities in the data, ask probing questions, and err on the side of a full investigation any time there is any reason to suspect even the slightest impropriety.

Reading the media accounts of the Atlanta cheating scandal, one is reminded of the Enron scandal, in which high level executives engaged in fraudulent accounting practices that devastated a company, its shareholders and employees right under the noses of its board of directors.  In response to such abuses, the federal government passed the Sarbanes-Oxley Act, which imposes a higher standard of accountability on corporate executives and boards of directors.  Among other things, the Act requires top management to certify the accuracy of financial reports.  The Act also imposes heightened responsibilities on corporate boards, through standing audit committees, to oversee the actions taken by top management.

Testing data is to schools, students and parents as financial information is to corporations, shareholders, and employees.  Perhaps a set of rules similar to those imposed by the Sarbanes-Oxley Act is necessary to ensure the integrity of student data.  Perhaps top management (such as the Superintendent, and chief accountability officer) should be required to personally investigate and certify the accuracy of student test data.  Likewise, perhaps school boards should be required to have standing committees to review the process and ensure its integrity.

Many argue that the lesson learned in Atlanta and elsewhere is that “high-stakes” testing will inevitably lead to cheating.  This is a sad conclusion, indeed.  It reflects a lack of confidence in the ability of all students to improve, and excuses from responsibility those adults upon whom our students are relying to help them improve.  Worse yet, it excuses those adults from even behaving ethically, and endorses behavior that sets a terrible example for students.  Cheating is not symptomatic of an inherent flaw in high-stakes testing (though there may be flaws).  Instead, cheating reflects a lack of integrity, leadership and good governance that is essential to the success of our system of public education.

There are very few areas in which school districts require more rather than less regulation.  Sadly, it appears that this may be one such area.  With or without additional regulation, however, the critical role of the school board in preventing such abuses cannot be overstated or overlooked.

— Kim Clark

---

[i] For further guidance on best practices in testing security, see “Issues and Recommendations for Best Practices,” which is based on comments and ideas generated during a Testing Integrity Symposium that the U.S. Department of Education held in February 2012.  This publication can be found at http://nces.ed.gov/pubs2013/2013454.pdf.

Securing Our Schools in the Wake of the Sandy Hook Elementary Tragedy — Pt I

The shooting last week at Sandy Hook Elementary School has prompted a great deal of debate across the country about gun control and access to mental health services.  The incident has also prompted increased scrutiny of school safety practices.  Of course, it is critical that schools review their lockdown procedures and other security measures on an ongoing basis, and ensure that staff members are well trained in those protocols.  School safety experts generally agree, however, that the security measures in place at Sandy Hook were appropriate and reasonable, and indeed saved lives.  Of course, all systems have limitations.  A criminal intent on breaking in at any cost will be difficult for any institution (other than a maximum security prison) to stop.  In fact, children are far safer in school than in other public places such as shopping malls, movie theaters, parks, playgrounds, etc.  And they are exponentially more likely to be killed in an auto accident than in an incident like the one that took place at Sandy Hook Elementary School.

Schools could increase police presence on campus.  Research indicates, however, that seeing armed police officers roaming the school can be scary for young children and undermine their feeling of safety and security. Moreover, those who criticize districts for spending too much on administrative as opposed to classroom expenses should be aware that school security, including on-campus police officers, is an administrative expense (which for many districts, is not insignificant).

Some, including Texas Governor Rick Perry, believe that allowing school personnel to bring guns to school is a valid solution.  They claim that a school employee with a gun who was properly trained could have stopped the perpetrator of the Sandy Hook shooting before he was able to kill so many people.   Statistically, however, it is far more likely that a legally purchased gun will be used not in defense of but against its owner or a member of his or her household (and by analogy the school it is intended to protect).

Even if this hypothetical gun-wielding employee turned out to be the James Bond in Governor Perry’s fantasy, i.e. capable of exercising good judgment and perfect accuracy under extreme pressure, allowing employees to bring guns into the workplace, and especially into schools, is a very bad idea.  The chance that most schools will ever experience anything like what happened at Sandy Hook is extremely slight.  Most schools, however, do experience some incidents of violence each year.  Add guns to this environment, regardless of who owns them, and the outcomes of those incidents are likely to be far worse.  It is a travesty that the perpetrator of the Sandy Hook shootings was able to gain access to legally purchased guns.  Locating more guns on-site and making them even more accessible will only escalate violence in our schools.  Moreover, if seeing police officers with guns on campus undermines children’s sense of safety and security, imagine what it would do to a child’s sense of security to receive a poor score on a homework assignment from a teacher packing heat.

If we want to invest in making our schools safer, we need to look at the areas of greatest risk to our students.  Fortunately or unfortunately, the greatest risk to students does not come from the outside.  The greatest risk comes from individuals students encounter on-campus with a colorable reason for being there.  That said, the one area in which many public school systems could be doing better is in conducting background checks of school employees, volunteers, contractors and others who come into contact with students on campus.  That subject, however, warrants a separate, more detailed discussion.  Accordingly, stay tuned for Part II, which will examine the ways in which some States’ and districts’ policies concerning background checks could be amended and/or supplemented to better protect students.  As for the adequacy of existing school security measures, and the suggestion that teachers be allowed to carry guns to school, please let me know you think.

The UPD blogger, Kim Clark is a senior consultant with UPD.  Prior to working with UPD, Kim served as the General Counsel for the Scottsdale Unified School District in Scottsdale, Arizona, as well as a labor and employment attorney at Steptoe & Johnson, LLP.

The Follower’s Manifesto

In my six years of teaching, I had plenty of colleagues who carried on non-stop private conversations through every faculty and department meeting they attended. The very educators who brought down the wrath of God on misbehaving or inattentive students became pouty, apathetic, or downright antagonistic when another adult had the gall to suggest that there was something these individuals needed to know or had yet to learn.

I know this mindset well, as I possessed it for a time:

“What does Vice Principal Smith know? He hasn’t been a teacher for 10 years…”

“I wish they’d let me get back to my classroom—I have so much to do and this is useless.”

“How could a consultant, who has never taught, possibly give me any advice about education?”

To be sure, some of this anger and indifference is well founded. I cannot count the number of faculty meetings I sat through where the principal read aloud (verbatim) from a schedule that affected 1/10^th of the school’s population. But to focus on this smaller point is to obscure a larger one: as much as we often hear that we lack good leaders in the education world, I believe the bigger problem is that we lack good followers.

Very few people have the privilege of holding a role in life in which they are consistently leaders, always laying out an agenda to be executed by those around them. Instead, most of us hold a more nebulous position—we are leaders of some and followers of others, and these roles change over time. Teachers are the perfect example of this—student achievement in the classroom requires great leadership on their part, but that leadership must be informed and supported through the following of administrative guidance, research-based standards of practice, community desires, and expert advice.  Yet while educational literature is rife with treatises on leadership (one of my primary introductory packets to Teach For America in 2004 was called Teaching as Leadership), there is little talk of following.

So what are the characteristics of a good follower, and how will they make a difference in education? With the help of the comparatively sparse followership literature[1], I’ve compiled this non-comprehensive list:

1. Good Followers are Open-minded. Too often in education, we assume that the best ideas for student achievement are contained in our own heads, or at the very least within our own dogma. We must be willing to adjust our approaches based on the advice, feedback, and new sources of information we receive.
2. Good Followers Disagree and Commit. Even good leaders will make decisions that their followers may not always agree with. This is perfectly reasonable, and followers should feel free to communicate that disagreement to leaders. However, once a decision has been finalized, followers must commit to act upon it as if it was their own. Refusal to act upon a decision prevents evaluation of the decision’s effects further down the road. This is the piece that I and my colleagues most often struggled with as teachers. It was easier to poo-poo a new administrative initiative about backwards planning for a million little reasons, than it was to buy into this initiative and change our ways.
3. Good Followers are Active Listeners and Collaborators. Listening to and participating in a conversation requires full attention and critical, collaborative thinking. The non-stop responsibilities of most jobs (especially teaching) can also function as excuses to mentally (or even physically) check out of one’s listening responsibilities. Grading takes precedence over listening to a department head, lesson planning replaces one-on-one time with a mentor. I know—I’ve been there. But I also know that listening and participating in collaborative opportunities is an important part of creating school culture and promoting practices that improve student achievement. It is through this collaboration that decisions are made and tested, and that leadership is held accountable.

With support from UPD’s Bob Pipik, Nick Goding, and (former employee) Dustin Odham, Highland Park High School in Topeka, KS has taken advantage of a federal grant to install a collaborative process of student and classroom data evaluation. Every progress report and grading period, teacher teams meet to examine trends in student attendance, grades, behavior, and test scores, both within their classroom and throughout the team. Students who are at risk are identified and intervened with as a team or individually using a “Student Tracker” created and molded through an iterative process of teacher and administrative feedback. This approach has led to a narrowing of the achievement gap between African American and White students, and has improved student test scores overall by almost 10 percentage points. And all of this has come as a direct result of attentive and excellent followership. It is true that school administration wrote the grant and initiated the data evaluation process (and for that they should be praised), but it was the school’s teachers who approached the process with an open mind, contributed to its functioning through collaboration with leadership, with outsiders (UPD), and among themselves during the teacher team meetings, and they have remained committed to its functioning for the past two and a half years.

It should be obvious that we can’t all be leaders all of the time, but that doesn’t mean we must resign ourselves to lives as desk jockeys, pushing paper for the man.  While my examples throughout this blog are based at the school level, the call for good followers is a universal one in the field of education (and beyond). Equity and excellence in public education will require that most of us make a commitment not just to lead, but to follow. From teachers to bureaucrats to consultants, we can shape and challenge our leaders, and the world around us, through our openness, our commitment, our action, our honesty. It’s time that “follower” stopped being a dirty word.

TM

---

[1] See Kellerman, Barbara. Followership: How Followers Are Creating and Changing Leaders. Harvard Business School Press, Cambridge, MA. February 18, 2008 as a prime example of the emerging field.

–Tim Marlowe

You should have been a doctor!

My doctor friends are always amused when education folks hold up the medical field as the exemplary profession. But medical envy is rampant across classrooms, offices, and the blogosphere, where the frustrated opine: “If only our society held teachers in the same regard as doctors…” and “If only teachers were paid like doctors…” and the politically-charged, “If only teachers had self-organized as professional associations, rather than adopt the industrial union model…” (see Rotherham’s blog post for a harsh snippet comparing the AMA and the NEA).

Another example – Dr. Atul Gawande’s book Better: A Surgeon’s Notes on Performance, which describes the challenges of increasing performance in the medical field, is now required reading in some graduate-level education policy classes.

Over in the teacher-prep side, a recent National Council for Teaching Quality (NCTQ) study compares teacher prep programs in Illinois, with pretty sobering results. The NCTQ work is similar to an early-20^th century study of medical schools – a study which contributed to the eventual shuttering of almost half of all medical schools due to abysmal performance.* Ben Carey also has an interesting take on preparation programs across sectors.

And coming up next month over in the data-driven part of town, Education Sector will tell us what can be learned from the medical field (not to mention Google and Farmville!) around data collection and use. Their seminar, Next Decade of Education Data takes place Dec. 7 in Washington, DC.

And so, readers out there – do you agree that the education field has much to learn from the medical field, especially around performance, preparation programs, and data?

*UPD is actually working with the NCTQ to bring this study national. More on that to come. (JF)

BEHAVE!

Anyone who is involved in establishing pay-for-performance compensation models for teachers and principals should spend a little time in advance reading up on Dan Ariely (The Upside of Irrationality), Daniel Pink (Drive) and other behavioral economists before embarking on a pay system based on the conventional wisdom about what motivates people. Unfortunately, most people, including several prominent superintendents, believe that money—in the form of better pay and performance bonuses—is the key to attracting higher quality teachers to the profession and motivating them to perform better.

I do believe that higher pay would attract more people to teach, though the relatively low pay for teachers compared to other professions is probably not as big a barrier to a better teacher talent pool than the filter of requiring a degree from a teachers college. (But that’s a topic for another blog.) Yet, the premise that once someone decides to become a teacher we still need to provide some sort of bonus structure to ensure that they bring their “A” game to the classroom is flawed for many reasons, but I’ll just tackle two of them.

The first has to do with why people get into the teaching profession in the first place, and it is not to make a lot of money. Ask any teacher why he or she became a teacher and the answer is typically about being inspired by a favorite teacher they had, wanting to give back to the community, an intellectual fascination with a particular subject area, or a desire to work with children and help them learn. If you got into teaching for any of these reasons or their many variations, you don’t turn it off because you’re not paid enough. There are deeper drivers at play, and if we don’t pay attention to the intrinsic motivations our teachers bring with them, we could actually do more harm than good when we set up pay-for-performance systems. As noted in Pink’s book Drive:

“Careful consideration of reward effects reported in 128 experiments lead to the conclusion that tangible rewards tend to have a substantially negative effect on intrinsic motivation. When institutions…focus on the short-term and opt for controlling people’s behavior, they do considerable long-term damage.”

So, if you inadvertently chip away at the intrinsic rewards teachers get from teaching—which is the main reason they enter the profession in the first place—how is that likely to impact classroom outcomes? (That’s a rhetorical question, in case you were wondering.)

The second flaw in the pay-for-performance premise has to do with how bonuses linked to high-stakes outcomes might negatively affect a teacher’s performance. In The Upside of Irrationality, Ariely describes several experiments that got at this issue. His conclusion is that moderate and high bonuses work well for tasks that are mundane, require little creativity or problem-solving, and are largely within one’s control. So, an incentive for a professional basketball player to make a higher percentage of free throws might work well, as might a bonus for higher productivity on an assembly line. But those types of tasks don’t come close to relating to what a teacher does in the classroom. And for tasks that require innovation, creativity and problem-solving, moderate and high level incentives actually make performance drop. As Ariely notes:

“[W]hen the incentive level is very high, it can command too much attention and thereby distract the person’s mind with thoughts about the reward. This can create stress and ultimately reduce the level of performance.”

None of this is to say that we shouldn’t link pay and bonuses for teachers and principals to performance. No one who has ever worked with us can think that we don’t support such accountability. But there needs to be much more nuance in setting them up than is occurring in most states and districts that are trying it. And what is likely to happen when they fail is that, by association, all pay-for-performance models will be tainted by their failure. (DA)