NAEP 2009 Math Results Show Little Progress for NYC Schools

Last week, math scores for 2009 were released for the eighteen cities now participating in the NAEP Grade 4 and 8 Trial Urban District Assessment (TUDA) component, released bi-annually as “The Nation’s Report Card.” Operated under the aegis of the U.S. Department of Education and reported by their National Center for Educational Statistics, the NAEP’s are the only math and reading exams administered across the fifty states (reading test results are expected in Spring 2010). Given on a statistical sampling basis without high stakes consequences, these exams are generally considered the single best measure of national educational progress.

The Math results for 2009 were not the best of news for NYC. Among the eleven large cities who participated in 2007, only Boston and D.C. showed statistically significant gains in Grade 4 and only Austin and San Diego in Grade 8. No city declined, but the rest, including NYC, showed either no gains or gains sufficiently modest that they couldn’t reliably be attributed to real gains (as opposed to random chance). For NYC, these results stand in sharp contrast to the rather sizable gains in Grade 4 and 8 Math reported over the last two years via the NYS Grade 3-8 annual exams (on which School Progress Reports, grade retention, principals’ bonuses, and the Mayor’s political persona are based).

As reported in the local media (Jennifer Medina in the NY Times, Rachel Monohan and Meredith Kolodner in the NY Daily News, Yoav Gonen in the NY Post, and Beth Fertig at WNYC.org), NYC showed a one-point gain in average exam score in Grade 4 (from 236 in 2007 to 237 in 2009) and a three-point gain (from 270 to 273) in Grade 8. In addition, the percentage of Grade 4 students judged Proficient or better on the NAEP exam rose from 34% in 2007 to 35% in 2009; for Grade 8, the numbers were 22% in 2007 to 26% this year. By comparison, the NYS exams showed NYC Grade 4 proficiency or better increasing from 74.1% to 84.9% and Grade 8 moving up an astonishing 25 percentage points, from 45.6% to 71.3%. Comparable numbers since 2003 are shown below.

Grade 4 - % Proficient or Higher (Level 3+4 for NYS exams, Proficient + Advanced for NAEP)

2003 --- NYS 66.7% --- NAEP 21%

2005 --- NYS 77.4% --- NAEP 26%

2007 --- NYS 74.1% --- NAEP 34%

2009 --- NYS 84.9% --- NAEP 35%

Grade 8 - & Proficient or Higher (Level 3+4 for NYS exams, Proficient + Advanced for NAEP)

2003 --- NYS 34.4% --- NAEP 21%

2005 --- NYS 40.8% --- NAEP 21%

2007 --- NYS 45.6% --- NAEP 22%

2009 --- NYS 71.3% --- NAEP 26%

The inflation in all the NYS-based numbers naturally suggests that NYS/NYC students’ mathematical knowledge is nowhere near what is claimed by NYSED and NYCDOE.

This hardly comes as a surprise to most; it largely reflects the inherent fallacy of the NYSED’s testing regime, whose high stakes under NCLB have been substantially amplified under mayoral control of NYC schools until test prep and teaching to a specific test have come to dominate math teaching in the city. Nevertheless, measuring NYC’ students’ mathematical progress against the NAEP’s does suggest that, at least at Grade 4, both sets of scores are moving upward in some degree of encouraging lockstop. The picture in Grade 8 is far different and far more gloomy, particularly viewed within the framework of the huge reported gains in 2008 (14 percentage points, to 59.6%) and 2009 (another 11.7 percentage points, to 71.3% as shown in the table above).

These latest NAEP results offer a highly reliable reality check on the claimed math gains of NYC public school students. At Grade 4, the NAEPs suggest that the NYS test results have a degree of validity with regard to progress, although the 50-percentage-point gap on achievement is a major contradictory indicator of mastery level. At Grade 8, the NAEP’s paint a truly worrisome picture of a 45-percentage-point achievement gap and almost no measured progress in six years on the NAEPs during a period when NYC results on State exams signal gains bordering suspiciously on miraculous.

More disconcerting still is the realization that the 2009-year Grade 8 students who took these latest NAEPs are the first "pre-high-school end products" of the NYC public school system under Joel Klein and mayoral control during the six-year period (2003 - 2009) since the TUDAs and mayoral control both began.

Michael Bloomberg’s campaign for his third mayoral term was predicated in no small part on his claims to be NYC’s “education mayor” as justified by NYS exam scores and a host of other DOE self-generated statistical measures. In my next posting, I will delve much more deeply into the NAEP’s raw data to compare NYC’s progress since 2003 against the nine other large cities who have participated in TUDA since that time and also reveal how many of the DOE’s (and Mayor’s) claims are simply not supported by the only truly independent measure of NYC public school education currently available.

More on the bogus school bonuses

All three dailies are onto the absurdity of the DOE granting huge bonuses to staff at low-performing schools that received "C" and "D" on their progress reports – including, according to the Daily News, at least one school that is closing.

The bonus program will eat up almost $40 million in precious education funds this year. Despite all the budget cuts already enacted at the school level, and more to come, leading to layoffs, larger classes, the loss of after school and enrichment programs , the DOE insists that these indefensible bonuses will be continued (NY1).

"It's always cited as one of the most novel, exciting programs. And I think New Yorkers want results. We are paying for results," said Schools Chancellor Joel Klein.

Yes, results based on inflated test scores, credit recovery, and widespread tampering with student test scores.

From the Post: “Somebody has to be complicit in this robbery -- it cannot be that DOE can simply appropriate public money as it sees fit," said Paola de Kock, whose son graduated in June from Stuyvesant HS in Manhattan.

Unfortunately, they can and they will continue to waste our taxpayer money -- as far into the future as the eye can see. Unless the City Council stands up for our children and stops them. Don't hold your breath.

See also the article in the Times.

Bloomberg and Klein's personal credit recovery program

See the presentation given by Klein to business leaders yesterday. Some news accounts here: Times, GothamSchools, NY1, Post, Daily News.

As usual, the Chancellor tried to take credit where no credit is due -- for big increases in test scores that occurred in the winter and spring of 2003, despite the fact that his policies were not implemented until the following fall. Perhaps these retroactive claims are Klein and Bloomberg’s personal credit recovery program.

In fact, DOE officials and the mayor were quite subdued when there were big test score gains in 2003. For more on this, see our blog here.

As the Times pointed out when the state ELA scores were first reported on May 21, 2003:

“The city's positive results come at a time when Mayor Michael R. Bloomberg and his schools chancellor, Joel I. Klein, are trying to overhaul the public school system and impose a uniform reading and math curriculum at all but the highest performing schools.

…Experts said the sharp increase in test scores could prove problematic for Mr. Bloomberg and Mr. Klein, since it is too early for them to take credit and sets a benchmark for next year that may be hard to match….

The higher scores, particularly at schools that have showed sustained increases since 1999, also gave new ammunition to critics of Mr. Bloomberg's changes, who said that they might do more harm than good by disrupting existing programs….

This is how the state math gains were reported when they were released on October 22, 2003:

“Fourth graders across the state made stunning gains in their math scores last spring, with even sharper increases in New York City…In the city, news of the gains, which were particularly pronounced in the Bronx and in some of the poorest-performing districts, elicited cheers among teachers and principals. But not everyone greeted the news so enthusiastically:

The suggestion that city schools were on the upswing put Chancellor Joel I. Klein, who is overhauling them, in a tricky position. While the chancellor’s critics pounced upon the higher scores as evidence that the school system did not need such an overhaul, some of his allies acknowledged that he would now be under even more pressure to show gains next spring.

A more accurate and fair analysis would look at data only from 2003-2008; instead of claiming credit for gains that occurred before the Bloomberg/Klein policies were put into place.

2.. None of these gains is meaningful given the evidence of the NAEPs, which shows no significant progress in NYC since 2003 in any grade or subject except for 4th grade math. We know that the state tests have been inflated, and there’s been terrific amount of test prep in NYC and teaching to the test, unrelated to any real gains in learning.

3. See slide 13: as of 2004, still only 55% of NYC eighth graders who make a 700 --level 3 – graduate in four years. (not even counting the discharge rates which would bring their actual graduation rate even lower. ) This is a huge repudiation of the quality of our high schools. Even the fact that only 87% of 8th grade students who hit level four, the highest level, graduate on time is pretty awful.

4.No info is offered on the results of the science and social studies exams – which are low-stakes tests, like NAEPs, and thus more reliable. Unlike math and ELA scores, for which the DOE offers financial rewards to teachers, schools, and kids, with little or no protections against cheating, these are the exams where we could expect less distortion in results.

NYC continues to have some of the lowest scores in the state in those subjects, even in the largely middle class districts. 28 of 32 districts in NYC were in the bottom 10% in the state in Science; 26 of 32 districts were at the bottom 10% in the state in Social Studies.

I’m sure there’s more deception and spin in these slides; please take a look and leave a comment on the comment page!

The absurd, silly, ridiculous school grades

See articles on this year’s elementary and middle school grades, or “progress reports”: Times, NY Post, Daily News, NY1, and this piece in Gotham schools (the best).



They are replete with examples of the absurd results, with schools such as PS 82 in the Bronx jumping from an “A” to an “F” and back this year to an “A”. PS 8 in Brooklyn which got an “F” last year and this year got an “A”, with the principal saying they did nothing different. The three schools that Klein tried to close last year to put charter schools in their place-- P.S. 194, 241 and 150 – all got “As”.

In fact, 84% of the schools got an “A” this year, and only two schools out of more than one thousand got “F”s. This is grade inflation that would put any human being other than Joel Klein to hide his head in shame.

And yet, according to the NY Times, “he clearly took pride in the results. “If you’re asking whether I would rather see less A’s,” he said, “the answer is no.”

Meanwhile, 87% of principals said in a recent survey that their schools were unable to provide a quality education because of excessive class sizes.

The absurdity of the grades this year derive from two profound flaws: First, 85% of the grade is based on one year’s gains or losses in test scores, which experts have found to be statistically unreliable and extremely erratic.

And two, the state tests have become so much easier and their scoring so lax that students can pass them without reading the questions – as long as they manage to fill in a few bubbles along the way. (For more on this scandal, well-reported everywhere except the Times, see the Daily News here and here, Gotham Schools and this NY Post column by Diane Ravitch.)

I wrote an oped for the Daily news about the new grading system when it was first announced in the fall of 2007: “Why parents & teachers should reject new school grades.” It starts out this way: “The new school grading system unveiled this week by Chancellor Joel Klein is a fiasco.”

But what do you expect when you had two guys in charge, Liebman and Klein, who are ignorant as to statistics, the unreliability of test scores, and even the larger goals of education?

Perhaps the best result is where we have now arrived: hopefully everyone realizes that the emperor has no clothes and they should ignore these silly grades, as they should have in the first place.

The NY Times response, and my reply

See the response from the NY Times editor below to my critique of their test score article yesterday – and my reply. Read from the bottom up. See also this front page story from the Times from November 16, 2007 – in which, in opposition to the paper’s current stance, it was clearly stated that there were no significant gains in any area but fourth grade math since the Bloomberg/Klein reforms took hold:

“The national scores also showed little narrowing of the achievement gap between white students and their black and Hispanic counterparts. The results for New York and 10 other large urban districts on the federal tests, the National Assessment of Educational Progress, paint a generally stagnant picture for the city, although there are gains in fourth-grade math.”

Why have they changed their story now? If you’d like to chime in, please do! You can ask for a correction at nytnews@nytimes.com; write a letter to the editor at letters@nytimes.com, or email the public editor at public@nytimes.com

Amazingly, the same testing consultant who was quoted in yesterday's NY Times as saying the state tests are “about as good as we can build them” is quoted in today's Gotham Schools that "“It’s hard to trust the data right now."

____ ____ _____ _____

Dear Ian:

Thanks for responding. The broad evidence that the state test scores are inflated is clearly relevant to the Times reporting here and was totally glossed over.

Not one independent expert was quoted as to this issue – only those who have a strong interest in claiming credit for improvements, including the head of SED’s test advisory committee, the head of the Regents and the Chancellor himself. This is not balanced reporting. In fact, the headline that refers to “critics” is rather strange, when not a single legitimate critic is cited.

As I’m sure Elissa is well aware, there are many prominent administrators, researchers, teachers and principals who believe strongly that there has been rampant state test score inflation in recent years. Not only are huge jumps in the scores occurring in nearly all districts and in all grades– the middle grade increases last year were especially unprecedented -- but as has been widely reported in the Daily News and elsewhere, the cut scores have been lowered each year. To leave this out of your story seems negligent at best, especially given the room and the time your reporter had to expound on this issue.

Moreover, the lack of progress on the NAEPs in NYC provides a natural audit of the reported rise in state test scores.

As to whether or not the 2003 increases could be attributed to the leadership of Bloomberg and/or Klein, you are surely right that “It is not incorrect to say that fourth grade reading scores rose after the mayor took over. You are correct that some of the biggest gains occurred before his major reforms took effect, but we are not incorrect.”

But perhaps you might have mentioned the latter statement in your story!

Or, if you are implying instead that the increases in 2003 could be legitimately due to the Bloomberg/Klein leadership, as the administration is desperate to claim, the NY Times seems to have flipped positions. As was widely recognized at the time, by your own reporting staff, the 2003 gains were quite uncomfortable for Klein et. al. when they first occurred.

Let’s take a little trip back in time.As Diane Ravitch has recounted:

“The Legislature approved mayoral control legislation in June 2002, and Chancellor Klein assumed office in August 2002. He spent the next six months conferring with consultants. He and the mayor announced their plans and their pedagogical choices in January 2003. At that very time, students in the city and state were taking the state tests. The chancellor had done nothing in the schools prior to January 2003 to raise student achievement. His reforms were introduced into the schools in September 2003. Thus, it is inappropriate for Dr. Bell-Ellwanger [or any DOE official, as they have recently claimed] to take credit for any gains registered on the state tests that were administered in January 2003.”

This reality was well recognized at the time, by both the Klein administration and the NY Times reporting staff.

As David Herszenhorn wrote in the Times when the state ELA scores were first reported on May 21, 2003:

“The city's positive results come at a time when Mayor Michael R. Bloomberg and his schools chancellor, Joel I. Klein, are trying to overhaul the public school system and impose a uniform reading and math curriculum at all but the highest performing schools.

City officials, who might otherwise have been jubilant about yesterday's results, offered a muted reaction, saying that the gains were not broad enough and that the school system as a whole was still failing at least half the city's children.;;

''Although we are pointed in the right direction, there is still a lot of work that needs to get done,'' said a deputy schools chancellor, Diana Lam. ''Even with these gains, we have one out of two students that are not fluent readers at the fourth-grade level.''

Experts said the sharp increase in test scores could prove problematic for Mr. Bloomberg and Mr. Klein, since it is too early for them to take credit and sets a benchmark for next year that may be hard to match….

The higher scores, particularly at schools that have showed sustained increases since 1999, also gave new ammunition to critics of Mr. Bloomberg's changes, who said that they might do more harm than good by disrupting existing programs….

And he quoted some of these dissenters, which at the time included Norm Fruchter and Randi Weingarten, as saying that the scores instead resulted from the reforms introduced by the previous Chancellor, Rudy Crew.

Of course, Klein and company are now quite eager to take credit for these 2003 gains – which as regards the 4^th grade ELA NAEPs, show the only significant improvements since Bloomberg took office. But this revisionist history is hard to square with your paper’s own past reporting.

Let's see how Elissa Gootman herself reported the state math gains when they were released on October 22, 2003:

“Fourth graders across the state made stunning gains in their math scores last spring, with even sharper increases in New York City…In the city, news of the gains, which were particularly pronounced in the Bronx and in some of the poorest-performing districts, elicited cheers among teachers and principals. But not everyone greeted the news so enthusiastically:

The suggestion that city schools were on the upswing put Chancellor Joel I. Klein, who is overhauling them, in a tricky position. While the chancellor’s critics pounced upon the higher scores as evidence that the school system did not need such an overhaul, some of his allies acknowledged that he would now be under even more pressure to show gains next spring.

''While I am gratified by the test results released today for fourth and eighth graders in New York City, I must emphasize that it is hard to tell the true significance of any one set of results in isolation,'' the chancellor said in a statement. ''We must always look at results in comparison over a number of years. Only through comparison can we truly measure the progress we're making.''

That’s true of course. And there has been no significant improvement in the NAEPs in the years since – except in 4^th grade math [in which there was a 25% exclusion rate].

Since the NAEPs are widely seen as the gold standard “audit” test, admitting that you didn’t see the need to report in any depth on NYC’s lack of progress on the NAEPs is like saying you can report a story on the DOE’s finances without seeing the need to mention any of the numerous audits from the State or City Comptroller’s office that reveal widespread waste on the part of DOE, including misappropriation of state class size funds, huge growth in private contracts with 20% going over budget, and repeated awarding of no-bid contracts, without registering or documenting their rationale.

But that’s another story…..perhaps you might look into the no-bid contracts for Agudath of Israel, with the amounts awarded the organization by DOE dwarfing what the Times’ political reporters wrote about today in relation to the Mayor’s use of his discretionary funds.

Thanks, Leonie Haimson

---

From: Ian Trontz [mailto:trontz@nytimes.com]
Sent: Monday, August 03, 2009 6:20 PM
To: leonie@worldnet.att.net
Subject: test score article

Ms. Haimson:

We appreciate the attention you've given to this story, as much as we've appreciated your being a source for us over a number of years.

We do not, however, think that your email points out any inaccuracies. It is not incorrect to say that fourth grade reading scores rose after the mayor took over. You are correct that some of the biggest gains occurred before his major reforms took effect, but we are not incorrect.

Indeed, there are many interesting and important ways to look at the NAEP results, but the NAEP comparison was one small part of this report; we could not devote unlimited space to it given the parameters of this article, which is about the city’s performance on state tests. We therefore included a very broad, basic summary of the NAEP results.

We also, unfortunately, do not have the most recent NAEP results, but you can be assured that we'll report them in full when we get them.

As for whether the tests themselves are deficient, I think the article discusses that, perhaps not with the depth you are seeking, but as with the NAEP, we cannot go down every road.

Ian Trontz, Assistant Metro Editor/Education

NY Times falls in line with the Bloomberg PR spin control

This New York Times article on the Bloomberg/Klein record on test scores is incomplete, biased, and in some cases, clearly inaccurate.

1- The article falsely claims that fourth-grade reading scores rose on the national exams called the NAEPs, whereas there has been no significant gain since 2003, when the “Children First” reforms were introduced.

2- As reported in our book, “NYC Schools under Bloomberg and Klein: What Parents, Teachers and Policymakers Need to Know,” there has been no closing of the achievement gap between racial or ethnic groups in any grade or subject tested by the NAEPs, considered by experts the "gold standard" of assessments.

3- The article fails to mention that in fourth grade math, the one area that did show a significant increase in the NAEPs, the student exclusion rate was 25 percent, far higher than any other city tested.

4- The article fails to mention that in 8^th grade reading, NYC made less progress on the NAEPs than any of the nine other cities tested.

5- The article fails to quote any independent experts or point to any evidence of test score inflation on the state exams, including lower cut scores, the repetition of similar questions, or the narrowing of topics covered, as revealed in articles in the Daily News and elsewhere.

Instead, the article quotes at great length their favorite expert, Howard Everson, head of a panel assembled by the State Education Department, who says that “New York’s tests were “about as good as we can build them,” as well as Merryl Tisch, the head of the Regents, and Joel Klein himself, all interested parties who predictably claim that the results indicate actual improvements in learning.

And though the system is full of teachers and even principals who will attest to the fact that the exams have become easier over time, the reporter chooses to quote not a single one.

Given the evident bias of this article, it appears clear that the Times has been captured by the Bloomberg/Klein PR machine, and can no longer be trusted to provide objective analysis of their education record.

If you’d like to chime in, you could ask for a correction at nytnews@nytimes.com; write a letter to the editor at letters@nytimes.com, and/or email the public editor at public@nytimes.com