Saturday, January 1, 2011

How well do above average American States do compared to above average countries in test scores?

Over at National Review Online, Reihan Salam has some complaints about my PISA-comparison.

First, he notes that even if European-Americans and Asian-Americans are doing well, minorities are not, and they are going to be a larger and larger part of the U.S population. This is completely true.

However my analysis tells us two new things:

* The problem is not likely to be the American public school system itself. If it was the education system that constituted the problem, Asian- and European-descended students (70% of the U.S population) would not be performing so well relative to other nations. After all, they are in the same government controlled system. Studies that use more sophisticated methods to control for school quality come to the same conclusion: It's not the schools.

* Europe has a similar achievement test gap between majority/minority. It's just that the size of the European minority population is still small, so it doesn't affect the national average as much.

But if your aim is to isolate the treatment effect of a policy (in this case the design of the school system), the size of the group impacted is irrelevant. Turks in Sweden have approximately the same gap with Swedish natives as Mexicans in California to California natives. This should conclusively end one discussion: the Scandinavian school system has not discovered how to solve the majority/minority achievement gap.

If you want to explain why Sweden has less aggregate economic and social inequality, the relative size of the minority groups in question matter. If you want to compare the efficiency of school systems, it doesn't. It is immaterial for what we are doing, a comparison of how well the school system closes the achievement gap, that Turks are less than 1% of the Swedish population and Hispanics 30% of the Californian population. It is sufficient to show that they have the same gap. If anything, European schools should be doing better considering the smaller size of the disadvantaged groups.

Jumping up and down in excitement, and congratulating Swedish and other Social Democratic school systems for having "solved" the achievement gap while sending fact finding teams there to learn from their system is ridiculous. Their system hasn't solved anything, the same gap is there, they just have few minorities so the gap influences national averages less and is less apparent to outsiders.

So under closer scrutiny it turns out that the "secret" of Scandinavia's lack of a achievement gap is that until recently, they didn't let many minorities into their countries. Once they started to, the achievement gap among socially disadvantaged groups emerged, a little smaller in magnitude but likely due to the same causes.

In the 1960s, a naive liberal in the U.S could be excused for assuming that Sweden's success was due to their education or welfare policy, and that Sweden had "solved" societal problems such as ethnic achievement gaps. Other countries could just copy whatever Scandinavia was doing in order to deal with their social problems.

Now that we have all seen footage from Scandinavian ghettos, as well as national statistics from Sweden and the rest of Europe confirming an achievement gap with minorities, it should be clear that this theory was wrong. So why are we still spitting on American public schools and congratulating Social Democratic ones for producing roughly the same outcomes?


2. Salam believes that the U.S should be doing even better, and asks about the performance of Greenwich (Connecticut).

"The NEA, and Tino, would have you believe that it is very, very impressive that Greenwich is treading water with Yazoo...."the United States, in terms of GDP per capita, is closer to being the Greenwich [Connecticut] of the OECD than the Yazoo City [Mississippi]."

He has two points here. First, he believes that with America's resource advantage, the U.S should be outperforming other countries even more. I might write more on this in detail later.

The resource gap of the U.S vs. Western Europe is not the same as that between Connecticut and Mississippi. As I have written here, Connecticut has 110% higher per capita income than Mississippi. Meanwhile the United States has "only" 36% higher per capita income than Western Europe.

If we instead of Mississippi compare Connecticut with the U.S average, it fits better. Connecticut is 35% above the U.S national average in per capita income, and the U.S national average is about 35% above Europe. In this sense the United States is equivalent of a (albeit much larger) 'Connecticut of the OECD', which is what Salam wrote.

Nationally, Connecticut as a whole scores a little less than 0.2 standard deviations above the U.S mean. Well, my analysis suggested Americans of European-origin score a little less than 0.2 standard deviations above Western Europeans in Western Europe.

So if this is the criticism, it turns out that the U.S educational advantage is approximately what the U.S economic advantage would predict.

I can't fault Mr. Salam for wanting more output for the input American tax payers are spending. However my analysis suggests that unlike conventional wisdom on the political right, public education spending is not just wasted. It appears successful in buying America better test scores, for each given demographic group, and of course with diminishing marginal returns.

Second, Salam is not sufficiently "impressed" by the average American advantage. Let's see if we can impress him.

The data I presented was national averages. In the case of the U.S, this was the average of rich cities such as Greenwich Connecticut and poor cities such as Yazoo Mississippi. When you aggregate a large group (200 million people) the variance goes down, as Greenwich and Yazoo cancel each other out.

This is important to keep in mind when comparing with small outliers such as Hong-Kong and Finland. Even if the U.S beats Europe on average, the best performing tiny part of Europe will likely beat the average of the U.S. What I was impressed by was the fact that one large group (all Americans from Europe) beat another large group (all Western European native born) by a sizable margin, as well as beating most of the small, top performing outliers.

But I would be happy to disaggregate to defend my argument.

PISA doesn't report the score of U.S states, because the sample sizes are too small. However there is another similar international test of math and science of 8-graders that does have data on individual states, TIMSS. They also have data on 2 U.S states, Minnesota and Massachusetts (both are similar to Connecticut in terms of test scores nationally, according to other data).

Remember that the United States scores relatively worse on TIMMS than PISA, especially compared to Asian countries, because PISA includes reading (which Americans are good at), whereas TIMSS is only a test of math and science.



Massachusetts and Minnesota each score higher than every single European country, although Finland is not included.

The achievement gap between Minnesota and Sweden is as large as the gap between Sweden and Bulgaria!


Keep in mind that this is the average of Minnesota and Massachusetts, including immigrants and minority students. If you were for example to break out only Asian-Americans in one of those states, the results are going to be stratospheric.

For fun, I also used a third large test of 8-graders (NAEP) which allows me to break out Asian-Americans in Connecticut, but requires more work to make it comparable with PISA. If Asian-Americans in Connecticut score as high relative to other Americans on PISA as they do on the NAEP relative to other Americans, they would beat every single Asian country in PISA. So would Asian-Americans in Texas and New York and several other states.



Note: this second graph relies on translating scores on one standardized test of 8-graders (NAEP) to PISA-equivalent scores, assuming that your group scored 10% (in standard deviation terms) above the American national average on the NAEP means your group would also score 10% above the American national average on PISA. I know that those readers who are resistant to my argument are going to nit-pick and are going to to be reluctant to accept the comparison.

However regardless if you buy my second graph, what is very important for readers to understand is that the first graph with TIMSS is not based on me doing any sort of adjustment of the data. It's simply the average of the scores, as reported by the Department of Education, on a test that is internationally standardized.

This doesn't definitively settle the debate about the quality of education system, because of biased selection of Asian immigrants. But I hope this example illustrates that affluent groups in the United States destroy the international competition, and are competitive even when compared to small rich countries such as Hong-Kong.

No comments:

Post a Comment