My Article on Teacher Value-Added Data Dumping in The Indypendent

John Tarleton asked me to write this a few months ago for The Indypendent but held it waiting for the court case to be decided. We had to make it tight for the print edition so I left a lot out. His excellent editing made it more readable. An even shorter version will run in the Jan. 17, 2011 print edition. And a reminder - help support the work of The Indypendent, which has done so much great work in reporting on the ed deformers and the resistance.

Note: We haven't had the time to add the links on some of the studies mentioned. Will try to update but if any of you find them send them along to normsco@gmail.com.

On the web: http://www.indypendent.org/2011/01/11/teacher-test-scores

Judge Rules in Favor of Releasing Teacher Test Scores; Data Dump Would Promote a Flawed and Cynical Method of Accountability

By Norm Scott
January 11, 2011 | Posted in IndyBlog | Email this article
 

Would you gauge the effectiveness of individual doctors by the percentage of patients who live or die under their care? Should firemen be held accountable when a building burns down? Should individual soldiers in Afghanistan be compared to each other on the basis of “success” or “failure” in controlling the Taliban in a given area?

Any effort to do so would spark a major outcry. But when it comes to teaching, there is a different standard.

On Monday Manhattan Supreme Court Justice Cynthia Kern ruled that the NYC Department of Education was obliged to release the names of individual teachers with “value-added” test score results that purport to measure teacher effectiveness. Judge Kern brushed aside arguments by the United Federation of Teachers (UFT) that the release of unreliable data would unjustly harm teachers’ reputations writing “there is no requirement that data be reliable for it to be disclosed.”

The data dump will affect more than 12,000 classroom educators in Grades 4 to 8. The UFT is expected to appeal. There is some irony here as it was the UFT that signed off on the use of value-added in the first place after Joel Klein promised the results would not be made public, while many skeptical critics in the union raised questions about that deal and warned it would turn into a disaster for teachers and the union.

Feeding Frenzy

If the value-added data is ultimately released, expect a feeding frenzy as teachers are judged and shamed on an individual basis in the media. The larger purpose of such a data dump by DOE would be to further erode public support for teachers and force their union to renounce a seniority-based system just as Mayor Michael Bloomberg and his new Schools Chancellor Cathie Black are talking about having to lay off thousands of teachers due to budget shortfalls.

Ironically, it was only six months ago that the NY State Department of Education revealed that years of test score advances by city students had turned out to be a mirage causing Bloomberg and his former Chancellor Joel Klein a good deal of embarrassment. No matter – the Mayor is ready once again to wield unreliable test scores as a political weapon and most media in this city have deliberately short memories, having all too often been active partners in attempts to eviscerate teachers.

The value-added approach is the latest attempt to undermine teachers, the teaching profession and the teacher union by measuring teachers based on the performance of their students on standardized tests from year to year.

Crucial backing for such initiatives has come from private foundations led by billionaires like Bill Gates and Eli Broad who assert that data-driven models in the private sector can be transferred to public schools.

Their dream of using data to accurately measure the effectiveness of individual teachers is rooted in a vision of the school as a factory in which teachers are assembly line workers and rising student test scores equals rising workforce productivity. At long last, value-added supporters claim, good teachers will be rewarded and the poor ones forced to improve at risk of losing their jobs.

In reality, value-added measures are seriously flawed. They don’t fully account for external circumstances such as poverty or family turmoil that can affect a child’s performance from year-to-year.  Nor can they account for the fact the same child can take tests on different occasions and under different conditions and the results will differ.

A study by Mathematica Policy Research done for the US Department of Education showed that one-fourth of average teachers will be mistakenly identified for special rewards while one-fourth of teachers who differ from average performance by three to four months of student learning will be overlooked.

A recent study by Sean Corcoran of NYU  demonstrated that the New York City teacher data reports have an average margin of error of 34 to 61 percentage points out of 100. The National Academy of Sciences has also warned of the potentially damaging consequences of implementing these unfair and inherently unreliable evaluation systems. Even the NYC Department of Education’s own consultants have warned against using data for teacher evaluation.

Perverse Incentives
Value-added measures can not only be in error but they provide incentives for teachers to manipulate scores by using large amounts of classroom time practicing for tests or engaging in various forms of cheating. To the extent a teacher cuts corners one year to deliver improved test scores, a student’s next teacher will face that much greater of a challenge to deliver similar or even better results.

Teachers under the gun of having their very lifelihood threatened will be very careful about working with troubled children who could drag down their value-added ratings. Accountable Talk” wrote about dealing with a request to take a class full of difficult students:

“I did something I am still not proud of. I quit,” Accountable Talk wrote. “No, I didn’t quit teaching. I just quit volunteering to teach the very children who needed me most. When my AP [assistant principal] asked me to take them on again (which he would not do unless he knew I’d been successful), I said no. This year, those kids are with another teacher who has difficulty just getting them to sit in their seats.”

One of the political goals of the value-added approach is to break teacher unity by pitting them against each other. The competitive, zero-sum logic of value-added also undermines the spirit of collaboration which is essential to them refining and developing their craft. If sharing tips with fellow teachers will help them improve their value-added rankings, is it prudent to reach out and help teachers you are competing with?

The downside of a value-added approach doesn’t faze leading proponents like Eric Hanushek, a Stanford economist who has written that teachers’ scores should be made public even if they are flawed.

Several news organizations including The New York Post, The New York Times and The Wall Street Journal have filed Freedom of Information requests for New York City teacher test score data with which the normally secretive NYC Department of Education has been eager to comply.

Still, it is wise to remember that not everything that counts can be measured and not everything that can be measured counts.

Norm Scott worked in the New York City public school system from 1967 to 2002. He publishes commentary about current issues in New York City public education at ednotesonline.blogspot.com.

Here are some news story links I copied from Gotham:

• A judge said the city can release teachers’ value-added ratings. (GS, Times, Post, DN, NY1, WNYC, WSJ)
• The teachers union is planning to appeal the release, so it won’t happen yet. (GothamSchools)
• The Post says union president Michael Mulgrew is wrong to appeal the judge’s ruling.

Spread the word:

 

For ten years, The Indypendent has printed truth in the face of power. With political and economic systems faltering, there is an opportunity for real change from the bottom up. But this means having a vibrant independent media. Consider supporting The Indypendent as a monthly sustainer, donating as little as $5 a month. Please visit indypendent.org/donate. 

Subscribe to the Indypendent!



Below the fold: A list of some of the great pieces on ed that ran in The Indypendent

More Highlights from the Indy's 2010 Education Coverage



"Taking the Public Out of Schools" by John Tarleton

"Inside Columbus High School" by Mary Heglar

"Experience Is the Best Teacher: Bronx School Fights to Save Building Trades Program As DOE Pushes College Prep Over Hands-on Learning" by Mary Heglar

"Think Globally, Privatize Locally: Education Is Under Attack Around the World" by Lois Weiner

"Teaching Under Assualt: Two Visions of Education Clash as Bloomberg Prepares to Layoff 6,400 Teachers" by Norm Scott

"Why Teachers Unions Matter" by Lois Weiner

"Ticket to Ride: Students Win Metro Card Fight" by Jaisal Noor

"Students at Tilden High School Win Last Chance for Diploma" by Jaisal Noor

"An Education at Any Age: A Boy from Baghdad and His Parents Navigate Different Ends of the NYC School System" by David Enders

"Learning the 3 C's: Competition, Corruption and Cheating" by Arthur Goldstein and Lucas Hilderbrand

"A Parent's Guide to School Involvement" by John Tarleton

"Education Rediscovered" by Stanley Aronowitz

"Queer Youth Embrace Fluid Identities" by S. Leigh Thompson

"School Closings Pushback Begins" by John Tarleton

"Experience Be Damned!: Education Department's Self-Inflicted Crisis Leaves More Than 1,000 Veteran Teachers in Limbo" by Marc Epstein

Check out Norms Notes for a variety of articles of interest: http://normsnotes2.blogspot.com/. And make sure to check out the side panel on right for news bits.

Bloomberg's double standard

Is the mayor guilty of a double standard, as he defends the performance of the sanitation workers and fire department, whose ability to fulfill their duties were hampered by the blizzard, and yet he continues to blame teachers for conditions out of their control, and is pushing to release the unreliable teacher data reports? Lynne Winderbaum makes the case.

No one should say that our mayor is not understanding of how unique challenges can affect the statistical measurement of one’s job performance. And so it was that I listened to Mayor Bloomberg explain with a bit of impatience and annoyance that the city’s performance in the wake of the snowstorm was not up to par because of a series of unique challenges.

He begged for understanding because, you see, there were a large number of city agencies and personnel involved, there were near white-out conditions, and hundreds of city buses and dozens of ambulances were stuck in the snow.

But the mayor should be aware that all that matters is the outcome, not the difficulties inherent to the job. The data shows that the average response time to structural fires in 2008 was four minutes 33 seconds. The average response time for medical emergencies in 2008 was four minutes 30 seconds. However, in this case, data released today show that the Fire Department had a 3-hour delay in response to critical cases, like heart attacks, and 12-hour delays for non-critical calls. A five alarm fire in Elmhurst raged for 3 hours when firefighters were delayed by the blizzard conditions.

Surely, firemen and EMT’s are to be judged “ineffective” when it comes to a response time so far below the city standard.

Extraordinary challenges notwithstanding, emergency responses to all calls should be within five minutes. It is incumbent on the news organizations, for the sake of our citizens, to FOIL a list of all firemen and EMT’s that were on duty during this time period and to identify them by name in the newspapers. The mere fact that response time data was influenced by so many factors beyond their control, as detailed by the mayor above, is no excuse to fail to reach or exceed the standard of response time expected by the city. It is also unfounded to excuse the longer response times from any engine companies who were impacted by the increased demands created by the closure of firehouses in their neighborhoods.

It is commendable that the FDNY receives the gratitude of the citizenry and the satisfaction of knowing they have saved lives and property. But these things are not measurable as are response times.

Perhaps there should be merit bonuses for the fastest responders to ensure that firefighters show more dedication to their work and our citizens’ welfare. Those who take on the most challenging conditions are no exception. Data is king and the statistics are the only objective way to measure the value of the workers.

It is incomprehensible that in the face of this disappointing data the mayor would excuse FDNY performance by saying, ““And I want them to know that we do appreciate the severity of these conditions they face, and that the bottom line is we are doing everything we possibly can, and pulling every resource from every possible place to meet the unique challenges…”

Oh wait. Nobody wants to privatize the Fire Department or find reason for it to be run by corporate interests who have scant experience improving performance in fire and medical emergencies. Never mind. -- Lynne Winderbaum, retired teacher

Update on FOILed backup for DOE claims as regards teacher data reports

Yesterday in court, the United Federation of Teachers argued that the DOE should not release the teacher data reports to the public, despite FOIL requests from media outlets, because the value-added methodology on which they are based are statistically unreliable, among other reasons, a point also made by many researchers, including Sean Corcoran of NYU. (See articles about the court case in today's Gotham Schools, NY1, Daily News, NY Times, and Post)

In February 2009, almost two years ago, I submitted a FOIL request to DOE for a number of items related to these reports, including the supposed "panel of technical experts" who had approved the DOE's methodology, according to the statement in the 2008 document, “Teacher Data Initiative: Support for Schools; Frequently Asked Questions":

“A panel of technical experts has approved the DOE’s value-added methodology. The DOE’s model has met recognized standards for demonstrating validity and reliability.”

When the FOIL was partially responded to fifteen months later, DOE admitted that this expert panel had not actually approved its methodology, and sent me a report in which the panel expressed grave doubts about its reliability.

Juan Gonzalez of the Daily News wrote about this here; I wrote about it and provided back up documentation here.

In its 2008 FAQ, DOE had also claimed that there was a research study that confirmed their approach:

“Teachers’ Value-Added scores from the model are positively correlated with both School Progress Report scores and principals’ perceptions of teachers’ effectiveness, as measured by a research study conducted during the pilot of this initiative.”

In February 2009, I also asked for a copy this "research study." Coincidentally,I just received yet another email from DOE today, informing me that this study is still not complete, more than two years after the above claim was made, and nearly two years since I filed my original FOIL. (see letter above).

FOILed documents show how the DOE dissembled regarding their Teacher Data reports

Several years ago, Chancellor Klein determined to use value-added methods to measure the achievement gains of teachers of English and math teachers in grades 4-8, by comparing the standardized test scores of their students to these students' test scores previous year.

Since then, numerous studies have shown how inherently unreliable this value-added approach is in estimating teacher effectiveness, including an analysis by Mathematica for the US Department of Education, showing that there is a 25-35% chance of misidentifying the worst teachers as the best; as well as a recent study by Sean Corcoran of NYU demonstrating that the NYC teacher data reports have an average margin of error of 34-61 percentage points out of 100.

Critiques from the National Academy of Sciences in their comments on "Race to the Top" program, and noted academics assembled by the Economic Policy Institute have also warned of the potentially damaging consequences of implementing these unfair and inherently unreliable evaluation systems.

Yet in 2007, Klein hired a consultant from Battelle to develop a mathematical model that took a few school and classroom factors into account, including aspects of student background, as well as the class size and the experience level of the teacher. (Smaller classes and greater teaching experience are the only two observable factors that consistently lead to more learning, and yet DOE officials consistently devalues both of them. They are included in the model nevertheless, apparently because the research is so clear on this. )

Battelle then devised a formula that the DOE then used to produce "teacher data reports" which would ostensibly measure the effectiveness of these teachers (See here, for a sample version of these reports.)

In October 2008, Chancellor Klein made an agreement with the UFT that the teacher data reports would not be used to evaluate teachers, but only to help them improve their instruction:

“…as a tool for schools and teachers to use for instructional improvement. They are not be used to evaluate teachers…. Principals have been and will continue to be explicitly instructed not to use Teacher Data Reports to evaluate their teachers…”

Klein later went back on this promise, and in February 2010, he instructed principals to consider these reports when deciding whether to give teachers tenure :

“Principals and superintendents will consider the performance of each teacher who is up for tenure more carefully than ever, weighing multiple factors including Teacher Data Reports, where available and appropriate.”

As Gotham Schools reported at the time: “Those teachers who fall into the bottom or top 25 percent of the rankings will be red-flagged, alerting principals that the DOE recommends giving them tenure or cutting them lose [sic] . In total, about 160 teachers will fall into that bottom percentile. “

Though Klein also originally agreed with the UFT to keep the individual reports confidential, as are most performance ratings , and to resist releasing them to the public even if FOILed, he has gone back on this promise as well.

Yet even back in the fall of 2008, when the reports were first provided to principals, it was clear to me and many others that DOE would eventually use them to evaluate teachers, as by nearly all accounts, they have little or no value to helping teachers improve.

I also thought (and believe to this day) it is critical that any model used to determine a teacher’s effectiveness and professional future should be made publicly available, and independently vetted by experts in statistics and testing.

After spending months of unproductive requests to former chief press officer David Cantor and Amy McIntosh, the head of the DOE “talent office”, asking for more information about the model used and evidence of its reliability, I decided to FOIL this information.

Here is an excerpt from my original FOIL request, dated Feb. 23, 2009, along with the partial DOE response that finally came in May 24, 2010, more than fifteen months later:

Dear FOIL Officer:

This is a request for records pursuant to the Freedom of Information Law ("FOIL"), Article 6 of the Public Officers Law. We hereby request disclosure of the following information concerning the Teacher Performance Data Reports:

1) The model specification used in the report to produce estimates of teacher effectiveness;

2) The sources of the data for class size at the classroom and school level over the last ten years;

After many months of delay, here is an excerpt from the DOE response, dated May 24, 2010:

"With respect to items one and two of your request, while certain records being released today may be responsive to these items…The model was not designed to ascertain the impact of class size or other classroom or school level variables…"

Nevertheless, the “draft” technical report from the Battelle consultant indicated a very substantial impact of class size on achievement in math:

“In math...the characteristics that had a negative impact were percent free or reduced price lunch and class size.” (see also Table 5.2 in the analysis.)

Teacher experience level also had a significant effect, in both math and ELA.

In December 2008, more than a year before I filed my FOIL, in a document supposed to allay teachers’ fears, entitled: “Teacher Data Initiative: Support for Schools; Frequently Asked Questions, DOE had claimed that an independent panel of experts had attested to the model's validity and reliability, writing:

“A panel of technical experts has approved the DOE’s value-added methodology. The DOE’s model has met recognized standards for demonstrating validity and reliability.”

.So in my FOIL I asked for more information about this panel, including:

3) The identity of the members of the "panel of technical experts" who approved the model and/or methodology of these Reports, as well as the times and locations in which these experts met to discuss these issues with DOE staff;

Yet in 2010, in response to my FOIL, the DOE contradicted their earlier claim:

" With respect to item three of your request, I have been informed that approval of the model and methodology rested with the DOE, and not with any “panel of technical experts.” As a consequence, I have been informed that there are no responsive records that will answer this aspect of item three."

Instead, DOE sent a list of names on a document entitled “Technical Expert Panel: Value-added Data for Teachers Initiative”, dated Sept. 25, 2007 with representatives from the various groups, including the UFT, academia etc.

They also sent a report from a subset of these individuals, entitled “Statement on the New York City Teacher Value-Added Model” dated August 29, 2008, written many months after the teacher data reports were first released, and months after the DOE had claimed that an independent panel had validated their reports.

This statement was written by Tom Kane, then at Harvard and now at the Gates Foundation; Jon Fullerton of Harvard; Jonah Rockoff of the Columbia Business School, and Douglas Staiger of Dartmouth College. Far from validating the reliability of DOE’s methodology, these men expressed numerous reservations and caveats about value-added approaches in general, and made the following points, among others:

"1) Test scores capture only one dimension of teacher effectiveness, and they are not intended to serve as a summary measure of teacher performance…

2) If high stakes are attached, there will be potential to game these measures by teaching to the test, selecting students, altering difficult-to-audit student characteristics, or outright cheating. …

3) To calculate expected test scores…there are likely to be additional factors not yet considered that influence student achievement. …etc. " (For the full document, click here.)

In the FOIL, I also asked:"Whether the members of this panel were paid for their services and if so, the source of these funds..."

DOE responded: "With respect to item four of your request, the DOE did not pay any members of the panel for their services. It is my understanding that some members of the panel were paid by the Fund for Public Schools (the Fund) for related research work. Consequently there are no responsive record to provide."

Even though Klein runs the Fund for Public Schools out of Tweed, the DOE claims that it is not a public agency and does not have to release its financial records to the public.

In its 2007 FAQ, DOE had also claimed that there was another document that potentially confirmed the accuracy of their approach:

“Teachers’ Value-Added scores from the model are positively correlated with both School Progress Report scores and principals’ perceptions of teachers’ effectiveness, as measured by a research study conducted during the pilot of this initiative.”

So I also asked for a copy of this "research study", as well as a few other items, none of which have been provided to this day.

I last heard from the DOE on August 12 and again on September 10, 2010, saying that the above "research study" is still not complete, nearly three years after the DOE had claimed it existed.

In any event, the DOE has now commissioned researchers at the University of Wisconsin to "update" their teacher data reports, apparently not satisfied with the earlier versions produced by Battelle.

For copies of all these FOILed documents, see the Class Size Matters website ; for more on the problems with the teacher data reports, see today's column by Juan Gonzalez, as well as articles in the Daily News, the NY Times, GothamSchools and the Christian Science Monitor.

John Allen Paolos on Tweed's ongoing innumeracy

Check out John Allen Paulos in today’s NY Times, author of "Innumeracy", about how the current obsession with data often gives us the wrong answers; and steers us in the wrong direction:

Unless we know how things are counted, we don’t know if it’s wise to count on the numbers … Consider the plan to evaluate the progress of New York City public schools inaugurated by the city a few years ago. While several criteria were used, much of a school’s grade was determined by whether students’ performance on standardized state tests showed annual improvement. This approach risked putting too much weight on essentially random fluctuations and induced schools to focus primarily on the topics on the tests. It also meant that the better schools could receive mediocre grades because they were already performing well and had little room for improvement. Conversely, poor schools could receive high grades by improving just a bit.

We are now entering the fourth year of the Tweed’s inherently flawed school “progress reports” or grading system.

Each year the formula has been significantly revamped because of the absurdity of the previous year’s grades, including this year’s grade inflation, in which 84% of elementary and middle schools got "A’s". If the authors of this system were to receive a grade themselves, it would be an "F".

The school grades are based 85% on the previous year's state test scores, which themselves have been widely derided as unreliable. The formula used has also been shown to unfairly penalize schools with large number of high-need special education students, despite the DOE's claim to fully control for the student population.

And, as Paulos points out, they are essentially "random" as they are based on only one year's worth of test scores.

Yet, inexplicably, the DOE refuses to conform to reason and alter the formula so that it is based on more than one year’s data; despite the fact that Jim Liebman promised at their inception to base them on three years’ worth of test scores.

Other troubling problems related to the way in which the grades also rely in part on survey results from teachers and parents. Recent articles in the Daily News have shown how several principals have pushed teachers into giving them favorable reviews; with the threat that otherwise, DOE may close their schools based on low grades. Parents also commonly report the same sort of pressure, either externally or internally imposed.

The school grading system also ignores critical but highly variable factors that differ widely among schools and yet are largely outside the their control, such as class size or overcrowding, which can work against increases in achievement.

This omission is unjustifiable, given the fact that DOE’s “teacher data reports” that Klein says should be used in tenure decisions include class size as a key factor, showing that even Tweed educrats recognize that class size is an important contributor to teacher effectiveness, and their claims otherwise are so much hogwash.

Yet the teacher data reports are themselves problematic; and their formula has never been publicly released. I submitted a FOIL for the formula more than a year ago, as well as the identity of the “independent” panel that DOE had claimed had attested to its reliability, and have still not received anything in return.

As the National Academy of Sciences has pointed out, in its comments to Secretary Duncan’s misbegotten grant program “Race to the Top”, no system for evaluating teachers on the basis of test scores has yet been established that is ready for prime time, given all the inherently complex and imponderable factors that go into test scores, particularly at the classroom level. Any attempt to implement such a program, they urged, should be carefully tested and independently vetted, because it could very well have unfair and damaging consequences, not just to teachers but to our kids as well.

We have already seen how art, science and music and untested subjects have been minimized in our children’s schools since the over-emphasis on high-stakes tests has been imposed; with weeks more spent on test prep and less on learning.

All parents should closely watch the evolution of the recent agreement between the New York teachers union and the state, to base 25% of teacher evaluation on state test scores and another 15% on “locally selected measures of achievement that are rigorous and comparable across classrooms."

We must hope that whatever formula is used is independently and publicly vetted, does not result in even more unfair and unreliable measures of performance, and does not lead to more testing that will waste time and money, serving no purpose except to diminish the quality of education that our children receive.

You can comment on the US DOE blog about the fallibility of basing teacher evaluation on test scores here.

Times article on Klein's campaign to fire teachers regardless of seniority provokes more questions than it answers

In yesterday’s paper, the NY Times writes about Joel Klein's campaign to have the legislature pass a law that would allow principals to fire teachers, regardless of their seniority.

Excerpt: In 2008, New York City began evaluating about 11,500 teachers based on how much their students had improved on standardized state exams. A Times analysis of the first year of results showed that teachers with 6 to 10 years of experience were more likely to perform well, while teachers with 1 or 2 years’ experience were the least likely.

This article confirms what all research shows, that experience leads to more effective teaching. In fact, there are only two objective, measurable correlatives to effective instruction: smaller classes and more experienced teachers, and yet the administration has done everything it can to prevent either one from taking hold in NYC public schools.


Yet the article glosses over or omits much critical information.

Why does Klein want principals to be able to fire teachers with more seniority? It is not because of their quality, or lack thereof, but because they cost more money.

Why would principals tend to fire more experienced teachers if they get the chance? Not because they are less effective, but because of the “fair student funding” scheme imposed by Klein, principals now have to pay for their higher salaries out of their limited school budgets, meaning they are forced to choose between higher class sizes and experienced teachers.

Why is it that given the similar squeeze on the police and fire budgets, no one in the administration is recommending that either the Commissioner of Police or Fire Department be able to fire staff regardless of seniority? Indeed, there would be huge public outcry if the administration proposed firing senior police officers or firefighters; even though in their cases, there is far less research to show their increased effectiveness.


Of course, no one would dare put into place a system where police captains had total control over the staffing in their precincts, and had to pay for it out a limited budget, regardless of changes in local conditions and/or spikes in crime. Or for all the police officers to be fired in a precinct to be replaced with newbies if the crime rate rose.

No, this is part of the concerted attack on the whole notion of professionalism in the teaching force, and an attempt to destroy anything (read the union) that might interfere with the administration’s free-market, deregulatory, pro-privatization education policies.

One more question: how did the NY Times get a hold of the teacher data reports, based on value-added analysis of student test scores, to allow them to do the analysis mentioned above? Weren’t they supposed to be confidential?

According to an email from Jenny Medina, the reporter on the story, the Times submitted a FOIL request last year and received the teacher data reports on the district level, without names attached. It allowed them to “do some analysis, albeit fairly limited.”

Yet it is astonishing to me that there is a system in place for the last three years, in which these reports (see sample to the right) are distributed to principals and teachers, and now the Times as well, yet no member of the public has been allowed to see or vet the mathematical model on which they are based. This is especially the case, as given the chance, principals will likely refer to these reports to determine who to lay off.

More than a year ago, in February of 2009, I FOILed for the value-added formula embedded in the teacher data reports; as well as the identity of the supposedly expert (but still secret) panel that had approved of its validity and reliability, and the DOE has still not provided this information.

Every few weeks, I get the same canned response from the DOE, that “due to the volume and complexity” of the requests they receive, as well as the need to determine whether any redactions are needed, additional time is required, and I that should expect a substantive response within a month. And then I get the same exact email a month later. So much for transparency!

What's especially dangerous about all this, of course, is that through the "Race to the Top" fund, Arne Duncan and the US Department of Education is pushing states to adopt similar schemes, with teacher evaluation, pay and tenure based on student test scores, without any independent vetting of the reliability of such systems.

In fact, the National Academy of Sciences issued a report last October, warning that these systems are not ready for prime time, and might do more harm than good if implemented on a broad scale. From their press release:

"Too little research has been done on these methods' validity to base high-stakes decisions about teachers on them. A student's scores may be affected by many factors other than a teacher -- his or her motivation, for example, or the amount of parental support -- and value-added techniques have not yet found a good way to account for these other elements...

From the NAS report itself:

In sum, value-added methodologies should be used only after careful consideration of their appropriateness for the data that are available, and if used, should be subjected to rigorous evaluation. At present, the best use of VAM techniques is in closely studied pilot projects. Even in pilot projects, VAM estimates of teacher effectiveness should not be used as the sole or primary basis for making operational decisions because the extent to which the measures reflect the contribution of teachers themselves, rather than other factors, is not understood. ....such estimates are far too unstable to be considered fair or reliable.

And yet little attention was given these vehement warnings of the nation's top academic experts in testing and statistics; with no mention in the NY Times or other national media, and no acknowledgement by the administration that their efforts to impose these models on the nation's school districts might be off track.

No, the motto of Joel Klein and Arne Duncan as well as their sponsors in the business community and the Gates Foundation continues to be: full speed ahead! And the reckless high-speed train of experimentation that threatens to run over our children's schools hurtles forward, without any end in sight.