Too little and much too late, the Times finally reports on the state test score scandal

In yesterday’s front page story, entitled "On NY School Tests, Warning Signs Ignored," the NY Times' account of the state test score scandal left its own deficient reporting conveniently off the hook.

Anyone who was paying attention knew at least as far back as 2007 that there was rampant test score inflation, primarily through articles by Erin Einhorn and other reporters at the Daily News. These articles, which themselves relied on analyses from testing experts like Fred Smith, revealed that the test score inflation started as early as 2002, with questions and scoring on the state exams becoming easier over time.

See this 2007 article on our blog by Steve Koss, relating the ingenious experiment done by Einhorn in which she gave the 2002 and 2005 math tests to the same bunch of children, with the results showing that the 2005 exam was much simpler, a fact also reflected in the changing "P" values of the questions. Or this follow-up Einhorn article, where leading testing experts called for an independent audit, which of course did not occur until three years later.

Where was the NY Times amidst all these revelations? Absolutely nowhere. Even now, the Times article omits any mention of the Daily News’ earlier exposes – which brought attention to this issue to the wider public – and instead recounts as somehow meaningful that a few individuals who supposedly had doubts about the apparent rise in test scores, like Pedro Noguera and Kathleen Cashin, didn’t directly mention them to Klein– as though he might otherwise not have noticed the evidence that was splashed all over the Daily News!

The article also offers a rather irrelevant story, relating how Joel Klein earnestly tried to convince the state to change its scoring system to use a value-added method instead, as though that would have addressed any of the problems regarding the score inflation. In fact, that might have made things worse, as indicated by the way in which the inflation led to 84% of all NYC elementary and middle schools receiving "A" last year, based primarily on the value-added method, falling precipitously to only 25% this year, when the state decided to reset the cut scores.

The article also gives Regent Merryl Tisch a pass, letting her have the last word, saying “We came in here saying we have to stop lying to our kids,” without mentioning that throughout the test score inflation period, she was Deputy Chancellor of the Regents, and yet reliably supported Bloomberg and Klein's claims of great improvement.

The Times itself had plenty of reason to know about concerns about the state test score inflation throughout this period but not only failed to report on it, but generally toed the company line.

On August 4, 2009, at the very moment when Bloomberg was pressing for the extension of mayoral control of the schools, and two years following the Daily news exposes, the Times published a credulous story that recounted the steep increase in state test scores and the apparent narrowing of the achievement gap, including this quote from Joel Klein:

“Mr. Klein, for his part, said he was confident that rising scores reflected real improvements. “No matter how you look at them,” he said, “the picture is one that shows that the city is making dramatic progress.”

In the article , Klein put more emphasis on the apparent rise in proficiency levels rather than scores, “saying that the pass-rate was the more critical measure because it indicated proficiency, an important gateway to success…Our job is to get all kids to basic proficiency and then continue to move them forward, and I think we can do that.”

Now, of course, Klein emphasizes the higher scale scores on the state tests, since the reported proficiency levels have dropped so dramatically.

To the degree that the Aug. 2009 Times article expressed any doubts about NYC's dramatic rise in test scores, it was only the possibility that the drive towards high-stakes accountability had led to excessive test prep-- not any of the overwhelming evidence that the tests were simply and the scoring more lenient.

Indeed, yesterday's article focused on the same set of concerns -- excessive test prep, the public release of prior exams, and the limited number of questions on the exams -- rather than any of the more damning findings, showing how the state had rigged the results with easier questions as well as lower cut scores, both of which would directly implicate the state in the fraudulent outcome.

As I wrote in August 2009 to the Times editor, Ian Trontz:

“… there are many prominent administrators, researchers, teachers and principals who believe strongly that there has been rampant state test score inflation in recent years. Not only are huge jumps in the scores occurring in nearly all districts and in all grades– the middle grade increases last year were especially unprecedented -- but as has been widely reported in the Daily News and elsewhere, the cut scores have been lowered each year. To leave this out of your story seems negligent at best, especially given the room and the time your reporter had to expound on this issue.”

I also pointed out that the article had wrongly attributed gains to Bloomberg/Klein on the NAEPs by giving them credit for the increase in test scores since 2002, even though their reforms had not begun until 2003. Here is how Trontz responded to this point:

"We do not, however, think that your email points out any inaccuracies. It is not incorrect to say that fourth grade reading scores rose after the mayor took over. You are correct that some of the biggest gains occurred before his major reforms took effect, but we are not incorrect."

I concluded at the time:

“Given the evident bias of this article, it appears clear that the Times has been captured by the Bloomberg/Klein PR machine, and can no longer be trusted to provide objective analysis of their education record."

Shortly afterward, Wayne Barrett wrote about the controversy in the Village Voice,

"The Times front page piece last week -- headlined "Gains on Tests in New York Schools Don't Silence Critics" -- failed to quote any real critics, but gave Klein six self-promoting paragraphs. It did bury a single questioning quote from two academics not known as critics of the test scores in the thirty-fourth paragraph, but the top of the story trumpeted success scores that would have silenced any critic. If, that is, they were true."

Two days after the NY Times article ran, on August 6, the NY State Senate voted to renew mayoral control without any checks and balances, essentially allowing Bloomberg to retain his stranglehold over our schools. One of the few Senators who voted against the renewal, Sen. Carl Kruger, argued that the achievement gains claimed by Bloomberg and Klein would soon be found to be fraudulent.

In response, the Daily News editorial page, as Bloomberg-sycophantic as all the other NYC major dailies, argued vociferously against Kruger's claims. In the piece, the News editors referenced the recent NY Times article (since they could not cite the far more explosive reporting of its own staff ):

“Good luck with that. He'll sooner find Sasquatch under Chancellor Joel Klein's desk. The New York Times double-checked test results and concluded this week that they showed "a steady march upward."

Indeed, Sasquatch was hiding in plain sight under Klein’s desk all along.

Just a few days after the Senate vote, on August 11, 2009, Meredith Kolodner of the Daily News pointed out that the cut scores had been set so low on the 2009 exams that a sixth grade student could pass the ELA just by randomly guessing, while 7^th graders had to get just one extra question right to pass. At Gotham Schools, Diana Senechal tried taking the exams herself and confirmed that in the 5^th grade ELA, 6^th grade math, and 7^th grade ELA exams, a student could indeed achieve a level 2 through random guessing .

What’s most scary is that according to the latest Times article, the state is now apparently going to keep the questions on the exams secret forever, supposedly to ensure that they can keep their difficulty level stable -- to guard against excessive test prep using old tests.

So in future, it will be impossible for another reporter like the intrepid Einhorn to test the same bunch of kids with exams from two separate years, to prove how much easier they have become. Or for any parent to examine how flawed the questions may be. (Remember Brownie the Cow, the absurd questions on the 2006 4^th grade ELA exam?) As though keeping the questions secret is the only method that can be used to keep standardized exams stable over time!

Meanwhile, companies like the College Board manage to release the SATs each year, and still are able to equate them, by holding back the ungraded questions (what the Times article calls “field test questions”). Why that could not occur in the case of the NY State exams is never explained by the article.

Sorry, NY Times, this article is too little and too late. Years before, when Bloomberg was pushing to retain unilateral control over our schools based upon these inflated test scores, the paper of record owed it to its readers to inform them of these issues, and yet utterly failed.

(See my critique at the time of their August 2009 article, NY Times falls in line with the Bloomberg PR spin control; and the response from Times editor, Ian Trontz: The NY Times response, and my reply.)

Wishful Testing in this week's New York magazine

Check out the just-published piece in NY Magazine called Wishful Testing, featuring the comments of Steve Koss, blogger here, and which analyzes the state test score bubble, Campbell’s law, the over-hyped Harlem Village Academy, and connects the dots.

Between this, the recent Robert Kolker piece on the national craze of scapegoating teachers, and features by Jeff Coplon on Eva Moskowitz’ chain of charters and school overcrowding, the magazine has shown itself to be most valuable in dissecting the Bloomberg/Klein mirage.

Especially as compared to the New Yorker and the New York Times Magazine, whose reporting on the subject has been execrable.

Read it and leave a comment on the website!!

Listen up, mayor and the NY Times

The Times ran an editorial last week, "Parents need to know", which claimed that NYC parents didn't understand how the weak state exams had been "shortchanging students" and that they did their "children no favor" when they disrupted a Panel for Education Policy meeting on Monday. This was after the PEP chair, David Chang, refused to let them speak in response to a long, deceptive power point that claimed, despite the collapse of the state test score bubble collapse, that the schools had made great progress.

The editorial also excused the state test score inflation by stating, "Weak state tests are a chronic problem throughout the country — one that education departments are only beginning to come to grips with." Oh please. The NY test score inflation has been obvious to nearly all objective observers since at least 2007, despite the fact that it conveniently allowed the mayor to claim great improvements during his campaigns for the renewal of mayoral control and re-election, illusions that were bought hook line and sinker by the mayor's allies on the Times and the other editorial boards.

The editorial ended by claiming that the schools have nevertheless been "narrowing the performance gap between white and minority students." Yet the gold standard, the national exams called the NAEPs, show no narrowing of the achievement gap in any grade or subject since the Klein regime began. Truly, the Times editors "need to know" and start living in the real world, for they, along with the chancellor and mayor, have lost all credibility, and should stop criticizing parents while displaying their ignorance of what's really going on in our schools.Our mayor, who famously said people could "boo him at parades" if people didn't like his educational policies, will likely see more booing of his Panel of Eight Puppets in the months to come, unless he wakes up and starts to listen.

So what's the story? Have NYC schools made progress in test scores, or not?

Today's Daily News reports on how NYC parents are reacting in the Bronx and in Manhattan to the revelation that the great progress claimed by Bloomberg and Klein in the past was a fraud: "The scores make you want to cry," said Monica Major, a Bronx parent and president of School District 11's Community Education Council. "For years, we've been bamboozled and lied to."

Or as as Diane Ravitch writes in an oped, "When the [state] scores were released, there was a sound of bursting bubbles across the state. What once were miracles turned into mirages.”

Yet in recent days, Bloomberg and Klein have tried to argue that the evidence is still strong that city students have made "dramatic progress" under their leadership.

First, they have repeatedly claimed that since the city's average scores on the state exams have been rising over time, the fact that the proficiency cut scores have now been drastically lowered is irrelevant.

Yet the city's average scores on the state exams are likely just as unreliable as its proficiency levels, since the questions on the test have gotten easier over time.

See an earlier posting, in which Steve Koss discusses a 2007 Daily News expose showing how the increase in "P" (or "probability) values on the field test questions indicated that the state tests had become simpler between 2002-2005. (The chart to the right was originally in the Daily News.)

To prove this, intrepid reporter Erin Einhorn even gathered a bunch of children to take both the 2002 and 2005 math exams, and out our 34 kids, 24 of them did better on the 2005 exam.

As one of them said, "the 2002 questions were more complicated than in 2005... In 2005, they kept it short, simple and sweet.”

Really, the only semi-reliable measure we have of student achievement over this period is the city's performance on the national exams known as the NAEPs. Klein has also made questionable claims in recent days in relation to the NAEPs.

Here is what he said to the Wall St. Journal: "In both fourth-grade math and reading, New York City now performs at the same level as the nation overall," the schools chancellor said."

Is this true? No. In no subject or grade does the achievement level of city students match that of the nation as a whole, including 4th grade reading or math; though the gaps are smaller in that grade than in 8th.

A more meaningful comparison would be to look at in the change in NAEP scores compared to other large cities from 2003-2009, to see whether there has been exceptional progress during the Bloomberg/Klein era .

First I looked at the change in the percent of students at or above basic over this period, in NYC compared to other large cities and to Atlanta, which has made exemplary progress.

Note that compared to other large cities, NYC has done a bit better on average in two categories and worse in two: 8th grade reading and 8th grade math.

In all subjects, however, NYC's gains have been far less than those of Atlanta, (which incidentally has neither mayoral control nor has it been awarded the Broad prize.)

I also analyzed NYC's record in 8th grade reading, to see how our schools have done compared to other large urban districts.

This is what I found: NYC comes in dead last among the ten cities tested since 2003 in this subject -- the only city to make absolutely no progress.

Then I looked at the achievement gap issue between racial and ethnic groups, which Bloomberg and Klein's repeatedly claimed to have narrowed over the past few years. At one point, Bloomberg even went before Congress and boasted that he had cut it in half. Is that true?



No, of course not; In NYC schools, according to the NAEPs, the achievement gap is still very substantial, in all subjects and grades, from 23 to 35 points depending on the category.

But have these gaps narrowed over time? Well, the black-white gap has shrunk by one or two points in 4th grade reading and 8th grade math, which is relatively small in relation to the overall size of the gap; and the gap has increased slightly in the other two categories.

In terms of the white-Hispanic gap, Bloomberg and Klein's record is even worse. In three out of four categories, the gap has increased; and it has grown much larger in 8th grade reading and math.

So, when you look at the NAEP scores, the progress made in NYC under Bloomberg and Klein compared to other large cities is only a little better in some areas, and in some areas, like 8th grade reading or the Hispanic achievement gap, far worse.

A "teachable" moment for us all

For parents who are understandably agonized about the steep drop in state test scores at their children’s schools, as reported by the NY State Ed Dept. and various news outlets today, this is what they should keep in mind:

• Since the city experienced a 34-39% decline in the percent of students scoring at proficiency; any school that saw a drop smaller than that is doing okay.

• However bad a school’s test scores, it is not the fault of teachers, principals, parents or the kids themselves; it is the fault of the people running the system. Their names are Bloomberg and Klein, and they should hang their heads in shame. For years, they have insisted that the state exams were more reliable than the NAEPs, and refused to make the real reforms that would improve our children’s opportunity to learn, like reducing class size. And since their value-added achievements which they insist should determine teacher pay and tenure are next to nil; they should be forced to resign.

• The state is also at fault by colluding in the fiction of improving test results for the past five years, all to make it seem as though their high-stakes accountability policies were producing results.

More than anything else, as Steve Koss writes below, today's revelations (which are not revelations to people who have been reading this blog) should teach us all that TEST SCORES ARE NOT EVERYTHING.



Our kids – and our schools –should excel in all areas, not just on standardized tests, and the more policymakers focus on test scores to the exclusion of all else, as has happened in recent years, the less real learning will occur in our schools and the more our children will suffer.



It's as though a doctor only weighed his patients, and decided on that basis alone whether they were healthy. Not only could the scale be defective, as occurred in this case; but even if not, the patient could be slowly dying, and he would never know.

Harsh lessons for the editors of the Daily News and NY Times

The Daily News editorial board finally gave up today, and admitted that the city’s big gains in state test scores over the Bloomberg era have been a vast mirage, in an editorial called Harsh lesson for N.Y.

They were apparently briefed by NY Commissioner Steiner on the findings of Harvard's Daniel Koretz, which, according to this editorial and numerous news reports, has found that almost half of NYC 8^th graders who score as proficient on the state exams will never graduate from high school; and three-quarters of those who do graduate need remedial courses before entering CUNY community colleges.

Yet the editors of the News have up to now been very willing victims of this deception – and a lapdog of every single false or exaggerated claim from the Bloomberg/Klein administration.

This, despite the warnings of many testing experts like Bob Tobias, Diane Ravitch, Jennifer Jennings, and Fred Smith that the standards had been lowered, and their own terrific reporting staff of Erin Einhorn, Meredith Kolodner, Rachel Monahan, and Juan Gonzalez, who, for at least three years, have revealed the numerous ways in which the state test scores have been systematically inflated over time.

Those of us at the NYC parent blog have also written about this extensively; see this entry from Steve Koss, analyzing an Erin Einhorn story from Sept. 4, 2007. Einhorn did an ingenious experiment with middle school students, showing how the math exams had gotten easier, and how SED and the DOE should have and probably were aware of this fact (I can’t find the original article online anymore.)

Let’s hope that the News editors will finally learn something from this “harsh lesson” and start showing some independence.

The NY Times also bears some of the blame. I have never seen an editorial in the Times that even mentioned the possibility of state test score inflation; and the paper hasn't yet reported on the Koretz' findings.

In August 2009, when Bloomberg was pressing for extension of mayoral control of the schools and his own re-election, the Times published a credulous story that recounted the steep increase in state test scores without directly quoting any of the skeptics; and also incorrectly used the DOE’s preferred date of 2002 instead of 2003 to claim improvements on the national exams called the NAEPs.

The article omitted any of the abundant evidence that the state exams and their scoring had become easier over time. (See my critique of their August 2009 article, NY Times falls in line with the Bloomberg PR spin control; and the response from Times editor, Ian Trontz: The NY Times response, and my reply. See also Wayne Barrett's take on our critique of the Times.)

What are the chances that now that Bloomberg has successfully won his battle to retain nearly unlimited control over our schools, and is in the midst of his third term, the editors of the News and the Times will apologize to their readers, and admit that the smell they’ve told us was roses was really an artificial chemical, successfully concocted to fool them? Don't hold your breath.