Update on FOILed backup for DOE claims as regards teacher data reports

Yesterday in court, the United Federation of Teachers argued that the DOE should not release the teacher data reports to the public, despite FOIL requests from media outlets, because the value-added methodology on which they are based are statistically unreliable, among other reasons, a point also made by many researchers, including Sean Corcoran of NYU. (See articles about the court case in today's Gotham Schools, NY1, Daily News, NY Times, and Post)

In February 2009, almost two years ago, I submitted a FOIL request to DOE for a number of items related to these reports, including the supposed "panel of technical experts" who had approved the DOE's methodology, according to the statement in the 2008 document, “Teacher Data Initiative: Support for Schools; Frequently Asked Questions":

“A panel of technical experts has approved the DOE’s value-added methodology. The DOE’s model has met recognized standards for demonstrating validity and reliability.”

When the FOIL was partially responded to fifteen months later, DOE admitted that this expert panel had not actually approved its methodology, and sent me a report in which the panel expressed grave doubts about its reliability.

Juan Gonzalez of the Daily News wrote about this here; I wrote about it and provided back up documentation here.

In its 2008 FAQ, DOE had also claimed that there was a research study that confirmed their approach:

“Teachers’ Value-Added scores from the model are positively correlated with both School Progress Report scores and principals’ perceptions of teachers’ effectiveness, as measured by a research study conducted during the pilot of this initiative.”

In February 2009, I also asked for a copy this "research study." Coincidentally,I just received yet another email from DOE today, informing me that this study is still not complete, more than two years after the above claim was made, and nearly two years since I filed my original FOIL. (see letter above).

FOILed documents show how the DOE dissembled regarding their Teacher Data reports

Several years ago, Chancellor Klein determined to use value-added methods to measure the achievement gains of teachers of English and math teachers in grades 4-8, by comparing the standardized test scores of their students to these students' test scores previous year.

Since then, numerous studies have shown how inherently unreliable this value-added approach is in estimating teacher effectiveness, including an analysis by Mathematica for the US Department of Education, showing that there is a 25-35% chance of misidentifying the worst teachers as the best; as well as a recent study by Sean Corcoran of NYU demonstrating that the NYC teacher data reports have an average margin of error of 34-61 percentage points out of 100.

Critiques from the National Academy of Sciences in their comments on "Race to the Top" program, and noted academics assembled by the Economic Policy Institute have also warned of the potentially damaging consequences of implementing these unfair and inherently unreliable evaluation systems.

Yet in 2007, Klein hired a consultant from Battelle to develop a mathematical model that took a few school and classroom factors into account, including aspects of student background, as well as the class size and the experience level of the teacher. (Smaller classes and greater teaching experience are the only two observable factors that consistently lead to more learning, and yet DOE officials consistently devalues both of them. They are included in the model nevertheless, apparently because the research is so clear on this. )

Battelle then devised a formula that the DOE then used to produce "teacher data reports" which would ostensibly measure the effectiveness of these teachers (See here, for a sample version of these reports.)

In October 2008, Chancellor Klein made an agreement with the UFT that the teacher data reports would not be used to evaluate teachers, but only to help them improve their instruction:

“…as a tool for schools and teachers to use for instructional improvement. They are not be used to evaluate teachers…. Principals have been and will continue to be explicitly instructed not to use Teacher Data Reports to evaluate their teachers…”

Klein later went back on this promise, and in February 2010, he instructed principals to consider these reports when deciding whether to give teachers tenure :

“Principals and superintendents will consider the performance of each teacher who is up for tenure more carefully than ever, weighing multiple factors including Teacher Data Reports, where available and appropriate.”

As Gotham Schools reported at the time: “Those teachers who fall into the bottom or top 25 percent of the rankings will be red-flagged, alerting principals that the DOE recommends giving them tenure or cutting them lose [sic] . In total, about 160 teachers will fall into that bottom percentile. “

Though Klein also originally agreed with the UFT to keep the individual reports confidential, as are most performance ratings , and to resist releasing them to the public even if FOILed, he has gone back on this promise as well.

Yet even back in the fall of 2008, when the reports were first provided to principals, it was clear to me and many others that DOE would eventually use them to evaluate teachers, as by nearly all accounts, they have little or no value to helping teachers improve.

I also thought (and believe to this day) it is critical that any model used to determine a teacher’s effectiveness and professional future should be made publicly available, and independently vetted by experts in statistics and testing.

After spending months of unproductive requests to former chief press officer David Cantor and Amy McIntosh, the head of the DOE “talent office”, asking for more information about the model used and evidence of its reliability, I decided to FOIL this information.

Here is an excerpt from my original FOIL request, dated Feb. 23, 2009, along with the partial DOE response that finally came in May 24, 2010, more than fifteen months later:

Dear FOIL Officer:

This is a request for records pursuant to the Freedom of Information Law ("FOIL"), Article 6 of the Public Officers Law. We hereby request disclosure of the following information concerning the Teacher Performance Data Reports:

1) The model specification used in the report to produce estimates of teacher effectiveness;

2) The sources of the data for class size at the classroom and school level over the last ten years;

After many months of delay, here is an excerpt from the DOE response, dated May 24, 2010:

"With respect to items one and two of your request, while certain records being released today may be responsive to these items…The model was not designed to ascertain the impact of class size or other classroom or school level variables…"

Nevertheless, the “draft” technical report from the Battelle consultant indicated a very substantial impact of class size on achievement in math:

“In math...the characteristics that had a negative impact were percent free or reduced price lunch and class size.” (see also Table 5.2 in the analysis.)

Teacher experience level also had a significant effect, in both math and ELA.

In December 2008, more than a year before I filed my FOIL, in a document supposed to allay teachers’ fears, entitled: “Teacher Data Initiative: Support for Schools; Frequently Asked Questions, DOE had claimed that an independent panel of experts had attested to the model's validity and reliability, writing:

“A panel of technical experts has approved the DOE’s value-added methodology. The DOE’s model has met recognized standards for demonstrating validity and reliability.”

.So in my FOIL I asked for more information about this panel, including:

3) The identity of the members of the "panel of technical experts" who approved the model and/or methodology of these Reports, as well as the times and locations in which these experts met to discuss these issues with DOE staff;

Yet in 2010, in response to my FOIL, the DOE contradicted their earlier claim:

" With respect to item three of your request, I have been informed that approval of the model and methodology rested with the DOE, and not with any “panel of technical experts.” As a consequence, I have been informed that there are no responsive records that will answer this aspect of item three."

Instead, DOE sent a list of names on a document entitled “Technical Expert Panel: Value-added Data for Teachers Initiative”, dated Sept. 25, 2007 with representatives from the various groups, including the UFT, academia etc.

They also sent a report from a subset of these individuals, entitled “Statement on the New York City Teacher Value-Added Model” dated August 29, 2008, written many months after the teacher data reports were first released, and months after the DOE had claimed that an independent panel had validated their reports.

This statement was written by Tom Kane, then at Harvard and now at the Gates Foundation; Jon Fullerton of Harvard; Jonah Rockoff of the Columbia Business School, and Douglas Staiger of Dartmouth College. Far from validating the reliability of DOE’s methodology, these men expressed numerous reservations and caveats about value-added approaches in general, and made the following points, among others:

"1) Test scores capture only one dimension of teacher effectiveness, and they are not intended to serve as a summary measure of teacher performance…

2) If high stakes are attached, there will be potential to game these measures by teaching to the test, selecting students, altering difficult-to-audit student characteristics, or outright cheating. …

3) To calculate expected test scores…there are likely to be additional factors not yet considered that influence student achievement. …etc. " (For the full document, click here.)

In the FOIL, I also asked:"Whether the members of this panel were paid for their services and if so, the source of these funds..."

DOE responded: "With respect to item four of your request, the DOE did not pay any members of the panel for their services. It is my understanding that some members of the panel were paid by the Fund for Public Schools (the Fund) for related research work. Consequently there are no responsive record to provide."

Even though Klein runs the Fund for Public Schools out of Tweed, the DOE claims that it is not a public agency and does not have to release its financial records to the public.

In its 2007 FAQ, DOE had also claimed that there was another document that potentially confirmed the accuracy of their approach:

“Teachers’ Value-Added scores from the model are positively correlated with both School Progress Report scores and principals’ perceptions of teachers’ effectiveness, as measured by a research study conducted during the pilot of this initiative.”

So I also asked for a copy of this "research study", as well as a few other items, none of which have been provided to this day.

I last heard from the DOE on August 12 and again on September 10, 2010, saying that the above "research study" is still not complete, nearly three years after the DOE had claimed it existed.

In any event, the DOE has now commissioned researchers at the University of Wisconsin to "update" their teacher data reports, apparently not satisfied with the earlier versions produced by Battelle.

For copies of all these FOILed documents, see the Class Size Matters website ; for more on the problems with the teacher data reports, see today's column by Juan Gonzalez, as well as articles in the Daily News, the NY Times, GothamSchools and the Christian Science Monitor.

Times article on Klein's campaign to fire teachers regardless of seniority provokes more questions than it answers

In yesterday’s paper, the NY Times writes about Joel Klein's campaign to have the legislature pass a law that would allow principals to fire teachers, regardless of their seniority.

Excerpt: In 2008, New York City began evaluating about 11,500 teachers based on how much their students had improved on standardized state exams. A Times analysis of the first year of results showed that teachers with 6 to 10 years of experience were more likely to perform well, while teachers with 1 or 2 years’ experience were the least likely.

This article confirms what all research shows, that experience leads to more effective teaching. In fact, there are only two objective, measurable correlatives to effective instruction: smaller classes and more experienced teachers, and yet the administration has done everything it can to prevent either one from taking hold in NYC public schools.


Yet the article glosses over or omits much critical information.

Why does Klein want principals to be able to fire teachers with more seniority? It is not because of their quality, or lack thereof, but because they cost more money.

Why would principals tend to fire more experienced teachers if they get the chance? Not because they are less effective, but because of the “fair student funding” scheme imposed by Klein, principals now have to pay for their higher salaries out of their limited school budgets, meaning they are forced to choose between higher class sizes and experienced teachers.

Why is it that given the similar squeeze on the police and fire budgets, no one in the administration is recommending that either the Commissioner of Police or Fire Department be able to fire staff regardless of seniority? Indeed, there would be huge public outcry if the administration proposed firing senior police officers or firefighters; even though in their cases, there is far less research to show their increased effectiveness.


Of course, no one would dare put into place a system where police captains had total control over the staffing in their precincts, and had to pay for it out a limited budget, regardless of changes in local conditions and/or spikes in crime. Or for all the police officers to be fired in a precinct to be replaced with newbies if the crime rate rose.

No, this is part of the concerted attack on the whole notion of professionalism in the teaching force, and an attempt to destroy anything (read the union) that might interfere with the administration’s free-market, deregulatory, pro-privatization education policies.

One more question: how did the NY Times get a hold of the teacher data reports, based on value-added analysis of student test scores, to allow them to do the analysis mentioned above? Weren’t they supposed to be confidential?

According to an email from Jenny Medina, the reporter on the story, the Times submitted a FOIL request last year and received the teacher data reports on the district level, without names attached. It allowed them to “do some analysis, albeit fairly limited.”

Yet it is astonishing to me that there is a system in place for the last three years, in which these reports (see sample to the right) are distributed to principals and teachers, and now the Times as well, yet no member of the public has been allowed to see or vet the mathematical model on which they are based. This is especially the case, as given the chance, principals will likely refer to these reports to determine who to lay off.

More than a year ago, in February of 2009, I FOILed for the value-added formula embedded in the teacher data reports; as well as the identity of the supposedly expert (but still secret) panel that had approved of its validity and reliability, and the DOE has still not provided this information.

Every few weeks, I get the same canned response from the DOE, that “due to the volume and complexity” of the requests they receive, as well as the need to determine whether any redactions are needed, additional time is required, and I that should expect a substantive response within a month. And then I get the same exact email a month later. So much for transparency!

What's especially dangerous about all this, of course, is that through the "Race to the Top" fund, Arne Duncan and the US Department of Education is pushing states to adopt similar schemes, with teacher evaluation, pay and tenure based on student test scores, without any independent vetting of the reliability of such systems.

In fact, the National Academy of Sciences issued a report last October, warning that these systems are not ready for prime time, and might do more harm than good if implemented on a broad scale. From their press release:

"Too little research has been done on these methods' validity to base high-stakes decisions about teachers on them. A student's scores may be affected by many factors other than a teacher -- his or her motivation, for example, or the amount of parental support -- and value-added techniques have not yet found a good way to account for these other elements...

From the NAS report itself:

In sum, value-added methodologies should be used only after careful consideration of their appropriateness for the data that are available, and if used, should be subjected to rigorous evaluation. At present, the best use of VAM techniques is in closely studied pilot projects. Even in pilot projects, VAM estimates of teacher effectiveness should not be used as the sole or primary basis for making operational decisions because the extent to which the measures reflect the contribution of teachers themselves, rather than other factors, is not understood. ....such estimates are far too unstable to be considered fair or reliable.

And yet little attention was given these vehement warnings of the nation's top academic experts in testing and statistics; with no mention in the NY Times or other national media, and no acknowledgement by the administration that their efforts to impose these models on the nation's school districts might be off track.

No, the motto of Joel Klein and Arne Duncan as well as their sponsors in the business community and the Gates Foundation continues to be: full speed ahead! And the reckless high-speed train of experimentation that threatens to run over our children's schools hurtles forward, without any end in sight.

Political pull by charter school operator exposed

Some parents have asked me why I am involving myself in such a controversial issue as charter schools, seemingly unrelated to class size. But I don’t think it is.

District public schools that have a higher concentration of high-needs students are losing classrooms, libraries and intervention spaces to charter schools, which is neither equitable or good policy.

This is especially damaging, given the fact that only three out of 13 district schools slated to lose space next year to charters have reached their mandated class size reduction targets. For more on this see here, my comments on the charter co-locations that were voted on by the PEP last night.

Yesterday, on Good Day NY, when asked why the Chancellor seems biased towards charters, I said I didn’t really know, but that I suspected that many of the charter school operators are receiving preferential treatment because of their political connections.

A perfect example is revealed in today’s column in the Daily News by Juan Gonzalez, and in the emails he FOILed between Eva Moskowitz and Chancellor Klein.

Not only did the Chancellor intercede repeatedly with his own staff to get her chain of charter schools more space, when she had already received more than the formula would allow, helped her recruit parents for her schools by giving her access to their names and addresses, and also appeared at numerous fundraisers and helped her raise a million dollars from the Broad foundation, explaining how politically useful she was in organizing thousands of charter school parents to support Bloomberg, the continuation of mayoral control and raising of the charter school cap.

As Klein wrote to Dan Katzir of the Broad Foundation, “she’s done more to organize parents and get them aligned with what our reforms than anyone else on the outside.”

In her emails, Moskowitz repeatedly refers to her “army of parents” and many of them were indeed at attendance last night in the PEP meeting, along with their kids, cheering and chanting in support of their expansion into district buildings – all of which were approved, except for one.

Click on the email above, and check out the others on the Daily news website here. You'll be amazed.

You can also check out my appearance on Democracy Now .