How should we evaluate policies?

My purpose in writing this post is to pick up on a debate we have had on this blog and some of us have debated on other forums in the context of the beginning of the new government. I don’t think any of us really believed that a beleagured government, knowing fully that it was on the way out and facing a major crisis, was really going to be a vehicle for serious economic policy change in Ireland at micro level. But the new government has the type of majority and popular appetite for reform that may never be seen again and it is a good time to start this discussion anew.

A key feature of an economic recovery should be a more intelligent policy design and analysis framework for assessing policy changes in areas such as health, social insurance and education. At least part of this will involve greater economic skills on behalf of existing staff, more public assesment frameworks and less direct political interference in the actual design and implementation of policies. Another part will come from a willingness to engage with developments in economics and economic methodologies, particularly those emanating from microeconomics, the half of economics that hitherto has mostly been ignored by Irish governments.

A technical question of sizeable importance is how we should best actually evaluate public policies and who should take responsibily for doing this. By public policies, I mean any action of the state, including direct provision of funds, legislation, the creation of regulatory agencies and so on. One approach relies on measuring relevant economic variables in micro-units and examining the causal effect of the change and the level of the change. Various quantities can be calculated, including the average effect of the change across the population of interest and the effect on people specifically affected by the policy.

Estimating these types of quantities is technically and practically complex. One solution that has been advocated for several decades by various researchers particularly in the US and has found new force among development economics researchers in particular is the use of Randomised Controlled Trials (RCTs) to examine the effect of policies. A large number of researchers have been advocating the use of RCTs in public policy, with their main putative benefit being that they allow the isolation of the causal effects of policy.

There has been somewhat of a backlash against the more evangelical view of RCTS. In particular, people like Heckman and Deaton have pointed out that the use of RCTS brings new problems to the table, including the potential for Hawthorne effects, the difficulty in externally applying results that come from model trials, the cost of implementing trials and so on.

It is worth the while of the new government to engage with this debate. We have talked before about the fairly limited engagement, for example, of the aid programme with developments in Economics and the international debate in development economics would certainly help sharpen the case for spending money in these areas. One thing that should be kept in mind is the potential for the rapid deployment of randomised trials in the development of the surface features of tax and social welfare policy. An increasing literature in behavioural economics is showing impressive effects of fairly cheap changes in things like the nature of forms used in different policies and default options. This is spawning an increasing literature, some of which is convincing. As the Irish government moves to policies such as automatic enrolment for pensions, it will be increasingly important to examine how the surface features of these policies will influence people’s behaviour.

A few things I would say that I believe are reasonable conclusions from the current debate:

1. Randomised Trials should not be overhyped but they have strong potential in specific areas and the government should examine this potential.

2. A greater focus on defining the outcomes of public policies and engaging in sufficient longitudinal tracking to enable assessment of these outcomes should characterise the main decisions made by the new government. The types of radical changes coming down the tracks to pensions and healthcare should be matched by an effort to increase the level of intelligence used to assess these changes.

3. While the development of an independent fiscal council is obviously a positive step, the new government should have greater engagement with developments in microeconomics, including a greater emphasis on peer-review of major policy evaluations.

4. A cultural norm needs to emerge placing greater pressure on government departments and agencies to provide solid rationales for large-scale policy changes. In the US, various legislation mandates the deployment of rigorous methodologies to assess policy changes.

5. Policymakers need to become more aware of the basic nature of modern economics. Many recent Irish policy documents are noted by the real failure to engage with the type of research that actually drives the fields under discussion e.g. the innovation taskforce document was particularly weak in its research foundations and strategic documents are routinely published that do not seem to draw in any way from what has been learned in the relevant literatures.

6. One potential solution would be the development of a fast-track economics group within the Irish civil service, something that Karl Whelan, Colm Harmon and others have spoken about. In my view, such a group would need to place a heavy emphasis on ability to synthesise economic knowledge on sectoral and specific micro-policy issues. This does not neccesarily mean recruiting outside of the existing service but it does mean breaking norms within the civil service and prioritising the development of a group who would be given a mandate to engage with the wider economics world. The lack of economic expertise in areas such as health, social protection and finance is genuinely frightening and damages the credibility of the state in a major way, something that will be increasingly exposed in a less opulent environment. The almost complete absence of anyone with a PhD Economics qualification from the Irish civil service is something that cannot continue.

Harmon on Policy Decisions

Colm Harmon has an article in the Times last week on changing the way policy decision are made in Ireland.

And Budget 2011 did make some bright moves – the changes in the transit taxes for air travel is framed like an experiment – change designed to have impact, reversible if it fails, embedded if it works. We need to get into the same thought-space on microeconomic policy on all domains.

Colm references the Travel Tax reduction in the budget. Basically, Minister Lenihan played the part of Solomon and instead of leaving the 10 euro flight tax as it was or abolishing it totally, he has reduced it to 3 euro for the next year when it will be reviewed. There is definitely a progression here over the usual way of doing policy, one that seems calculated to call the bluff of Ryanair in particular. Of course, as an experimental design, it is going to be a tricky one to look at. Macroeconomic conditions are changing rapidly and this is a once-off change. Worth talking through for the econometrically minded of the group.

Teacher Voices: The VIVA Project

The VIVA Project is a new initiative that gives teachers an opportunity to collaborate, share ideas, and inform education policy and reform conversations at the state and national levels. VIVA stands for Voices, Ideas, Vision, Action.

The Goal of the VIVA Project is to identify ideas and opinions straight from the classroom, work together to create actionable policies to improve public education for classroom teachers and their students, and deliver them directly to U.S. Secretary of Education Arne Duncan.

Teachers are encouraged to share their ideas in both the national and New York idea mines. Register today! And check out its Facebook page, too. You have until October 10, 2010 to add your voice to the conversation.

Other notables initiatives in a similar vein include Teach PLUS, the Teacher Leaders Network and the Hope Street Group's Policy 2.0.

The VIVA Project is a worthy addition to an education reform marketplace that too often ignores and discounts the thoughts and ideas of actual classroom teachers. Kudos to Secretary Duncan for being willing to listen.

Check out the embedded video to learn more about the VIVA Project:

'Education Does Not Begin Or End At The Schoolhouse Door'

A lot of us in education policy get lost within our own locus of control. In my case, it's all about teachers. After all, teacher quality is the strongest school-based indicator impacting student outcomes.

Sounds good, right? Yes, and no.

While it is inevitable that one focuses on what one can control professionally, it is important to have a sense of the bigger picture. That goes for us in education. After all, research has shown that the influence of schools on student outcomes pale in comparison to family and social factors outside of schools' direct control -- especially, but not only, in the early years of childhood. So while it is critical that we concentrate education policy efforts around attracting, developing and retaining effective teachers, we also must attend to a variety of factors outside of schools that impact students' ability to learn and succeed.

This report (Promoting A Comprehensive Approach to Educational Opportunity) from Cross & Joftus, funded by the Mott Foundation, provides an important reality check to our typical tunnel vision. It also provides a series of recommendations to better coordinate a largely fragmented web of federal programs focused on children. It reminds policymakers and high-level government managers -- who have responsibility for interdisciplinary public policies -- that they need to think holistically and work in concert.

There are existing organizations and movements afoot -- Broader, Bolder, the Centers for Disease Control and Prevention's Coordinated School Health Program, the Coalition for Community Schools, National Assembly on School-based Health Care, The Rural School and Community Trust, come to mind -- that take such a broader view of education and what it takes to fuel student success.

Some excerpts from the Cross & Joftus report:

The dominant assumption of American educational policy is that schools, by themselves, can fully overcome the impact of social and economic disadvantage on children’s development into thriving citizens.

The ... No Child Left Behind Act (NCLB) ... perpetuated and gave further credence to the assumption that schools could fully mitigate the impact of low socioeconomic status on students’ achievement and that schools were also the chief cause of poor performance.

First, since at least 2000, there has been a broad scientific consensus that “virtually every aspect of early human development, from the brain’s evolving circuitry to the child’s capacity for empathy, is affected by the environments and experiences that are encountered in a cumulative fashion, beginning in the prenatal period and extending throughout the early childhood years.” As James Heckman, a Nobel Prize economist, wrote, “Life cycle skill formation is a dynamic process in which early
inputs strongly affect the productivity of later inputs [especially schools]. Put another way, “education” does not begin or end at the schoolhouse door, and the “education” that children receive before they enter school significantly affects their success after they go through that door.

Second, the evidence does not support the view that the substantial gap closing that had occurred by the mid-1980s was entirely the result of schools, though schools did indeed contribute.

Third, despite the ongoing debate about whether or not schools alone can level the education playing field, the federal government has long been engaged in a schools-plus approach.

Read Deb Viadero's blog post at Inside School Research on the study as well.

Image courtesy of University of Nebraska-Lincoln Extension

My Guv'nah Is Stronger Than Your Guv'nah

Governors are an interesting group. Always. They are not interchangeable spirits. Just think of some of the characters and personalities amidst their ranks: Arnold Schwarzenegger, Rick Perry, Bill Richardson, Jennifer Granholm, Ed Rendell, Haley Barbour.

With regard to education, governors do not come to the job with equal chances to impact the policy agenda. I grew keenly aware of this when I worked for the National Governors Assoiciation (NGA) from 2001 to 2004. Some of this is due to personalities and individual capacities, such as whether they effectively use their bully pulpit and engage in policy conversations. And some is due to politics, such as whether they campaigned for office on education. But much of the reason for this variation is out of governors' control: It is due to widely varying nature of state educational governance systems.

This Education Commission of the States brief [summarized below] maps four models of state educational governance, present in 40 of the 50 states. (The other 10 states utilize hybrid models, furthering confusing the situation.) The most important fact is that ONLY 13 governors directly appoint the chief state school officer. That gives one pause in considering how empowered chief executive officers really are to tackle changes to public education. Most certainly cannot go it alone - and perhaps that's a good thing in certain ways, but it certainly doesn't produce direct reform trajectories.

Model One: The governors appoints the members of the state board of education. The state board, in turn, appoints the chief state school officer (variously called the State Superintendent, Commissioner, Education Secretary, etc.) Twelve (12) states utilize this model: Alaska, Arkansas, Connecticut, Florida, Illinois, Kentucky, Maryland, Massachusetts, Missouri, Rhode Island, Vermont and West Virginia.

Such state systems do not provide the governors much power over education governance. They accrue it over time as they appoint state board members -- usually with staggered terms -- and eventually gain a majority if they remain in office long enough.

Model Two: In this model, the state board of education is elected and the board appoints the chief state school officer. Eight (8) states utilize this model: Alabama, Colorado, Hawaii, Kansas, Michigan, Nebraska, Nevada and Utah.

Clearly, this model generates extremely weak gubernatorial control over public education, although chief executives in these states still wield the power of the purse, vetoes, and the like.

Model Three: In this model, the governor appoints the members of state board of education. The chief state school officer is elected. Eleven (11) states utilize this model: Arizona, California, Georgia, Idaho, Indiana, Montana, North Carolina, North Dakota, Oklahoma, Oregon and Wyoming.

Again, this is a governance blueprint for weak gubernatorial influence, although right-to-work states with histories of strong state influence over education -- such as North Carolina -- challenge this general assumption. Former NC Governor Jim Hunt has a lot to do with this, I believe. In his case, the power of personality transcended a weak governance structure. Differences also can be caused by differential continuums of power between state boards of education and chief state school officers.

Model Four: In this model, the governor appoints the state board of education and the chief state school officer. Nine (9) states utilize this governance model: Delaware, Iowa, Maine, New Hampshire, New Jersey, Pennsylvania, South Dakota, Tennessee and Virginia.

This would appear to be a template for strong gubernatorial control over public education, but of course it doesn't always turn out that way, depending on personalities, political choices made, and state education systems with a strong history of and preference for local control (here I'm thinking Iowa, Maine, New Hampshire). However, this group of states has certainly produced recent governors that were strong leaders in education -- Tom Kean of New Jersey, Tom Carper of Delaware, Mark Warner of Virginia, Dick Thornburgh of Pennsylvania are examples.

The remaining 10 states function under modified versions of the above four models. Models A (Louisiana and Ohio) and D (Texas) are relatively strong pro-governor structures, while Model B empowers state legislatures over governors in New York and South Carolina. Model E as implemented in Minnesota and New Mexico also provides those governors with significant power; not so much in Wisconsin, although the Badger State governor historically has had very strong veto powers. (Ever heard of the Frankenstein or Vanna White veto?)

A. Elected and Appointed State Board; Appointed Chief

In Louisiana, eight board members are elected and three are appointed by the governor. In Ohio, 11 board members are elected, while the governor appoints eight members.

B. Legislature Appoints State Board; Appointed or Elected Chief

In New York, the state legislature appoints the board members and the chief state school officer is appointed by the board. The South Carolina legislature appoints the board, but the chief is elected.

C. Joint Appointment of State Board; Appointed or Elected Chief

The governor, lieutenant governor and the speaker of the House appoint members to the state board in Mississippi. The state board appoints the chief state school officer.

In the state of Washington, the board of education is made up of 16 members ­—­ five of whom are elected by district directors (three for the western half of the state, two for the eastern); one at-large member elected by members of boards of directors of state-approved private schools; the superintendent of public instruction; seven members appointed by the governor; and two student members (non-voting). The chief state school officer is elected. Washington moved from a model whereby the state board was elected by district directors (local boards) to this model in January 2006.

D. Elected Board; Governor Appointed Chief

The governor appoints the chief state school officer who also serves as the executive secretary of the elected state board. Texas uses this model.

E. No State Board or Advisory Only; Elected or Appointed Chief

Minnesota and Wisconsin do not have a state board of education. New Mexico has an elected body (Public Education Commission), but is advisory only.

Minnesota and New Mexico – chief state school officer is appointed by governor

Wisconsin – chief state school officer is elected.

As Education Week's Alyson Klein reports, in this recent blog post about the just-completed NGA winter meeting, governors of both parties are AOK with the Administration's initial movement on ESEA reauthorization. NGA Chairman and Vermont Governor Jim Douglas, however, did invoke the word 'flexibility,' which is a tried-and-true part of the NGA mantra and which is being peddled far more aggressively by the NGA's sister organization, the National Conference of State Legislatures.

In today's meeting, which was part of the National Governors Association's Winter Conference, governors voiced "zero" concerns about federal intrusion in state business when it came to the Title I proposal, Secretary Duncan said in an interview with reporters outside the White House.

"This is being lead by the governors," he said. "We have to educate our way to a better economy. All of the governors understand this."

That's all well and good. But, the fact is, that some governors' opinions matter more than others, and some, while not wholly irrelevant, are hardly decisive actors. The fact that Race to the Top has empowered governors to take the lead in education reform conversations and to lead states' applications for these competitive dollars has changed the dynamic somewhat. Because they are not directly in charge of public education in most states, however, most governors cannot expedite change along the lines that the Obama Administration is calling for without attending to building relationships, cajoling, convincing, and achieving reform one step at a time.

A Sociologist's Place in Educational Reform

Cross-posted from Brainstorm

On Friday the Sociology of Education section of the American Sociological Association held a small conference in San Francisco specifically focused on the role of the sociologist in educational reform. Organized by some of the section’s smartest young thinkers including Mitchell Stevens, Amy Binder, and Elizabeth Armstrong, the meeting was refreshingly thought-provoking. Not everyone in attendance was one of the usual suspects—for example, Tom Toch appeared to give a great talk on school reform.

Central to the day’s discussions was a topic near and dear to my heart: Can, and should, a sociologist of education conduct relevant educational research and try to have an impact on educational reform? Is the academic’s place in the academy, or in the schools? Even if a professor desires to become involved with policy and practice, is her voice welcomed? Considered? Or, as so many (but not all) seemed to suggest, are those efforts a waste of time given that economists appear to dominate policy discussions in ways we can't compete with? Are we simply better off sticking to addressing the "how and why" questions, leaving those questions of greater immediate importance—questions of causal impact, for example—to those who are professionally rewarded for applied research? Sociologists who want tenure, the more senior folks tended to say, need to bring education to sociology—to make contributions to their discipline. Others argued for the sociologist to focus on bringing that perspective to education—making contributions to educational reform.

Obviously the debate is moot if only one approach merits tenure—if the latter kind of work isn’t rewarded, those doing it cannot remain in the academy. So right now, it's most common for sociologists to make the academic work the center of their agenda, and do the more applied stuff on the side—like a hobby. But is it time for this to change? Can, and should, more applied sociological research on education be rewarded in the tenure and promotion processes? I can report there’s very little consensus among my colleagues in this regard, and that differences of opinion are not entirely explained by professional or generational status. However, what’s most remarkable is how impassioned grad students, assistant profs, and tenured professors all are about this issue. Strong opinions abound—and a willingness to engage in debate pervades. That, in and of itself, is exciting.

ps. If you'd like to read a graduate student's perspective on what transpired at that meeting, I encourage you to check out Corey Bower's blog (see the post I'm linking to, as well as ones before and after it).

(Re)Focusing on What Matters

Last week I spoke at a meeting of the Lumina Foundation’s Achieving the Dream Initiative, a meeting of policymakers from 15 states all working to improve the effectiveness of community colleges. At one point, a data working group shared results of its efforts to create new ways to measure college outputs. This was basically a new kind of report card, one capable of reporting results for different subgroups of students, and enabling comparisons of outcomes across colleges. Something like it might someday replace the data collection currently part of the IPEDS.

While it's always gratifying to see state policymakers engaging with data and thinking about how to use it in meaningful ways, I couldn’t help but feel that even this seemingly forward-thinking group was tending toward the status quo. The way we measure and report college outputs right now consistently reinforces a particular way of thinking-- a framework that focuses squarely on colleges and their successes or failures.

What’s the matter with that, you’re probably wondering? After all, aren’t schools the ones we need to hold accountable for outcomes and improved performance? Well, perhaps. But what we’re purportedly really interested in—or what we should be interested in—is students, and their successes or failures. If that's the case, then students, rather than colleges, need to be at the very center of our thinking and policymaking. Right now this isn't the case.

Let’s play this out a bit more. Current efforts are afoot to find ways to measure college outcomes that make more colleges comfortable with measurement and accountability--and thus help bring them onboard. That typically means using measures that allow even the lowest-achieving colleges at least a viable opportunity for success, and using measures colleges feels are meaningful, related to what they think they’re supposed to be doing. An example: the 3-year associates degree completion rates of full-time community college entrant deemed “college ready” by a standardized test. We can measure this for different schools and report the results. Where does that get us? We can then see which colleges have higher rates, and which have lower ones.

But then what? Can we then conclude some colleges are doing a better job than others? Frankly, no. It’s quite possible that higher rates at some colleges are attributable to student characteristics or contextual characteristics outside an individual college (e.g. proximity to other colleges, local labor market, region, etc) that explain the differences. But that’s hard to get people to focus on when what’s simplest to see are differences between colleges.

It's not clear that this approach actually helps students. What if, instead, states reported outcomes for specified groups of students without disaggregating by college? How might the policy conversation change? Well, for example, a state could see a glaring statewide gap in college completion among majority and minority students. It would then (hopefully) move to the next step of looking for sources of the problem—likely trying to identify the areas with the greatest influence, and the areas with the most policy-amenable areas of influence. This might lead analysts back to the colleges in the state to look for poor or weak performers, but it might instead lead them to aspects of k-12 preparation, state financial aid policy, the organizational structure of the higher education system, etc. The point is that in order to help students, states would need to do more than simply point to colleges and work to inspire them to change. They’d be forced to try and pinpoint the source(s) of the problems and then work on them. I expect the approaches would need to vary by state.

Don’t get me wrong, I’m not trying to absolve any college of responsibility for educating its students. What I’m suggesting is that we think hard about why the emphasis right now rests so heavily on relative college performance—an approach that embraces and even creates more insitutional competition—rather than on finding efficient and effective ways to increase the success of our students. Are we over-utilizing approaches, often adopted without much critical thought, that reify and perpetuate our past mistakes? I think so.

Image Credit: www.openjarmedia.com

Stricter charter oversight in MN

As a charter school board member in Minnesota, I have to say, it's about time.  

According to a story in today's StarTrib,

Legislative changes affecting the special breed of public schools will increase oversight, close loopholes and clean up unclear language that had made it easier for some schools to get away with sloppy management or outright theft. Charter school sponsors will have stricter guidelines -- which could drive some away -- and the state will have more power to withhold taxpayer money or to shut down a school that breaks the law.

The minutes of our board meetings are public record, so I don't feel guilty about saying that I think we could use a lot more oversight, or at least training, in our financial decision-making. We don't have anything illegal going on like the charter school director who walked away with $1.4 million, but it just sort of feels like the blind leading the blind. Apparently our authorizer has barely come around to check on us except to make token yearly visits - that is until we got audited this year and the state had to get involved.

In New York this never would have happened, because the system is much more centralized.  There are only two charter authorizers in the state of New York, and their standards are very rigorous.  I've heard terrible stories about some of the many Minnesota charter authorizers.  

So apparently the state is putting pressure on authorizers: 

Some authorizers have gone "above and beyond the law" watching over the finances and operation of their charter schools, but others rarely even visit schools, said Chas Anderson, state Education Department deputy commissioner. "Without a strong authorizer, it's really hit or miss whether a charter school is going to be successful."

Now, all of a sudden, our authorizer has started to pay attention.

Obama On Education

There are two stories in today's papers on Senator Obama's plans for education as the Democratic presidential nominee. (Has Clinton conceded yet?)

The first comes from the Washington Post. While the story is framed broadly around domestic policy and the headline ("On Policy, Obama Breaks Little New Ground") has a clear negative connotation, it offers up a hint of what's to come on education now that the campaign can see the light at the end of the tunnel:

Heather Higginbottom, who runs Obama's policy office at the campaign's Chicago headquarters, cited education as one area in which Obama offers ideas that are not traditionally Democratic, arguing that the problem is not all about schools or funding, but about parents who let their children watch too much television. She said his proposal to give teachers bonus pay if they receive special training or if their students score high on standardized tests is an idea that some liberal-leaning teachers unions oppose. And she said the campaign has brought "fresh thinking" on many issues, particularly on one of Obama's favorites: increased government transparency.

But Higginbottom said the campaign's emphasis is on practical solutions, not ideological points. "I know it's interesting from a political perspective to look left, right and center, but we want to put forward ideas that will move forward in Congress," she said. "And we have the potential to engage people in a way they haven't been engaged recently and give them the tools to participate."

David Axelrod, Obama's top political adviser, said that the campaign will devote more staff members to policy (there are now seven) and that the senator's speeches will increasingly highlight his proposals.

"The next six months is going to be about competing visions for this country," he said. "Obama is looking forward, and his policies will reflect that."

The second story comes courtesy of the Denver Post's coverage of an Obama campaign stop in Thorton, Colorado yesterday.

Obama used the town-hall event to tell about 400 people about his plans for education reform. He promised to fix the "broken promises" of No Child Left Behind, make math and science instruction a national priority, and encourage every child to learn a foreign language to better compete in a global economy.

While praising the goals of No Child Left Behind, Obama has criticized the program for not providing the funding to make it successful.

"We also need to realize that we can meet high standards without forcing teachers and students to spend most of the year preparing for a single, high-stakes standardized test," he said.

The Illinois senator said he would simplify the application process for financial aid for children going to college and give a $4,000 tax credit for students attending public universities and colleges. In return, the students would be required to do 100 hours of public service a year.

He also vowed to create a Service Scholarship program to recruit talented people into teaching and place them in overcrowded districts or struggling

"I will make this pledge as president to all who sign up: If you commit your life to teaching, America will commit to paying for your college education," he said to cheers.

For more on Obama's education policy plans, check out his official campaign web site here.

NY Times: "U.S. to Require States to Use a Single School Dropout Formula"

Boring as it may sound, this is actually really good news (and it's about time). As someone who's had to deal with a lot of graduation and dropout data, I know there are a lot of schools and families who are going to benefit from this.

From the article: The requirement would be one of the most far-reaching regulatory actions taken by any education secretary, experts said, because it would affect the official statistics issued by all 50 states and each of the nation’s 14,000 public high schools.

Margaret Spellings: Much Better than Rod Paige.

Wednesday morning blahs

These kinds of news articles always annoy me because they perpetuate the idea that there is only one kind of "history" and one kind of "literature." That there's this "common core" (hence the organization's name) of knowledge that students must have in order to be well-educated.

Grr.

Value

This blog used to be ALL ABOUT education policy, and then I became a teacher and stopped having the energy to follow it. But I really need to respond to what's going on in NYC with the teacher-level test score analysis.

First of all, we knew this was the direction they were heading a while back when they started assigning individual ID numbers to each student, the kind that can be used for value-added assessment. However, in order for that to work, the students need to be taking the type of test that was designed for value-added assessment, such as the TerraNova. The New York State tests, to my knowledge at least, were not designed for value-added assessment. They weren't even really designed to show gains from one year to the next for each individual student, though that's what the city would like to do with the data.

I'm not opposed to analyzing and publicizing "value-added" data, given the appropriate measuring tool. But I think whatever experiment they're cooking up right now could not possibly provide meaningful insight into whether this is a system that can work in NYC. It's like using a teaspoon to judge a 100-meter dash.