What Do Students Learn in College?

They might attend their lectures, do additional hours of study and get good grades; but... what are students learning in college? A recent article in the New York Times asks just this question: "what are America's kids actually learning in college?". Thanks to Peter C for sending on the link. According to the NYT article, "a provocative new book, Academically Adrift: Limited Learning on College Campuses, makes a strong case that for a large portion of the nation’s seemingly successful undergraduates, the years in college barely improve their skills in critical thinking, complex reasoning and writing." The authors are Professors Richard Arum of New York University and Josipa Roksa of the University of Virginia. A Google-Books preview is available here.

Arum and Roska's research draws on survey responses, transcript data, and the Collegiate Learning Assessment, a standardised test administered to students in their first semester and then again at the end of their second year. More details on the authors' research project are available here. According to the authors' analysis (of more than 2,300 undergraduates at twenty-four institutions), 45 percent of these students demonstrate no significant improvement in a range of skills, including: critical thinking, complex reasoning and writing (during their first two years of college). According to the authors, "many students come to college not only poorly prepared by prior schooling for highly demanding academic tasks that ideally lie in front of them, but — more troubling still — they enter college with attitudes, norms, values, and behaviors that are often at odds with academic commitment."

In relation to academic commitment, education economists would draw attention to the idea of non-cognitive skills. Pedro Martins has shown that a targeted program related to non-cognitive ability – study skills, motivation, and self-esteem – can improve student achievement at state schools in Portugal. In work that I have done with Liam and Colm, there is a clear demonstration of academic advantage for Irish university students who are more future-orientated and more conscientious. A targeted program (providing academic and social supports) was evaluated in the setting of an Irish university by Denny, Doyle, O’Reilly and O’Sullivan (2010); the results suggesting it was an effective means of improving academic outcomes for socio-economically disadvantaged students attending Irish universities.

Returning to the NYT article, it interprets Arum and Roksa's research to indicate that "students are hitting the books less and partying more. Easier courses and easier majors have become more and more popular. Perhaps more now than ever, the point of the college experience is to have a good time and walk away with a valuable credential after putting in the least effort possible." A study by three Cornell economists found a large increase in enrollment in courses with a median grade of A; after a new policy which introduced the publication of course median grades on the Internet. This is suggestive of students enrolling in classes where it easier to obtain higher grade-scores; an issue which I raised before in relation to the Irish Leaving Certificate: the final examination at the end of second-level education Ireland.

Recent research by Brian Jacob, Brian McCall and Kevin Stange (all from the University of Michigan) specifically addresses the issue of the consumption-value of college education. Jacob, McCall and Stange report that students "appear to value several college attributes which we categorize as consumption because their benefits arguably accrue only while actually enrolled, including college spending on student activities, sports, and dormitories." In addition, research by Babcock and Marks (discussed on this blog before) documents that academic time investment among full-time college students fell from 40 hours per in 1961 to 27 hours per week in 2003. Another article from the NYT tells us that Arum and Roksa's research is "consistent with the findings of the National Survey of Student Engagement, which has polled more than 2 million students at more than 1,000 colleges and universities over more than a decade — and reported that many spend little time studying or writing."

According to this article in the Huffington Post, the Collegiate Learning Assessment "has its share of critics who say it doesn't capture learning in specialized majors or isn't a reliable measure of college performance because so many factors are beyond their control." Nonetheless, it looks like this debate is set to grow more widespread in the future. It also resonates with last year's debate in Ireland about the issue of grade inflation. Introducing a research component into the undergraduate curriculum could be one way to improve students' skills in critical thinking, complex reasoning and writing. In 2005, Trinity College Dublin launched a summer programme offering undergraduate students, typically from Science and Engineering, the opportunity to work in research for an 11 week period, on projects in laboratory environments.

Does Drinking in College Affect Students' Grades?

I recently discussed on the blog new evidence pointing towards a positive relationship between students' grades in college and their later-life health outcomes. A recent economics paper adds a new dimension in this area by producing findings on the effect of preceding health-risk behaviour on subsequent academic performance. The paper is by Scott E. Carrell, Mark Hoekstra, and James E. West; and is called "Does Drinking Impair College Performance? Evidence from a Regression Discontinuity Approach". Abstract: here. Paper: here. Discussion on the Freakonomics Blog: here.

The paper exploits a discontinuity in drinking at age 21 at a college in which the minimum legal drinking age is strictly enforced. A comparison is made between the grades of students who turned 21 before final exams to those who turned 21 just afterward. The authors find that drinking causes causes statistically and economically meaningful reductions in academic performance, particularly for the highest-performing students. The authors suggest that concern regarding the harmful effects of drinking in U.S. colleges is reflected by the Amethyst Initiative - in which 135 university presidents and chancellors argue that current policy has resulted in binge-drinking by students.

Do Higher Grades Mean Better Health?

The relationship between education and health has been discussed several times on this blog, most recently by Kevin. Kevin mentioned that there is a positive socioeconomic gradient: more educated people have better health on average. However, this does not imply that one causes the other. Kevin also mentioned recent research by Nils Braakmann, which examines the relationship between education and health, exploiting exogenous variation from a compulsory schooling law in the UK. Braakmann's results indicate "neither an effect of education on various health related measures nor an effect on health related behaviour, e.g., smoking, drinking or eating various types of food."

An interesting way of moving this research topic forward could be to focus more on quality of education rather than quantity of education. While there are many ways to consider how best to measure quality, one outcome that differentiates between students is grade point average, or grades. This metric has been the source of much debate (some of which I mentioned yesterday), but there is already some research which has set about examining the relationship between grades and health. (As an aside, readers may be interested to know that there is a positive relationship between college grades and subsequent earnings, as reported by McIntosh (2006), Loury and Garman (1995), Jones and Jackson (1990), Filer (1983) and Wise (1975).)

The research examining the relationship between grades and health came to my attention via an article by Roni Caryn Rabin in the Health section of the New York Times. The article refers to research done by Dr. Pamela Herd, and her colleagues at the University of Wisconsin in Madison. The research uses the Wisconsin Longitudinal Study, which has been following more than 10,000 people who graduated from Wisconsin high schools in 1957. "Those students who finished in the top 25 percent of their high school class were healthier, decades later, than the ones who finished in the bottom quarter... Even among those who each had 12 years of education, the person who performed better had better health".

Dr. Herd considered the possibility that better non-cognitive ability (or certain personality traits) could be driving both academic performance and health behaviour. NYT journalist Rabin provides more detail on Dr. Herd's thoughts on the matter:

"One explanation is that the same psychological characteristics that make for a hardworking student — like conscientiousness, dependability, good study habits and following the rules — also shape healthy behaviors. But when Dr. Herd examined personality surveys the graduates of 1957 filled out, controlling for variables like family background and childhood health, she didn’t find a strong correlation with health status... She’s convinced there’s something about the actual mastering of academic material that’s vital... the process of developing critical thinking skills and improving cognitive function."

What Do Grades Really Mean?

The Christmas Day edition of the New York Times carried an interesting article entitled "A Quest to Explain What Grades Really Mean". The motivation for the article was based around concerns relating to grade inflation, a topic which has been discussed before on this blog; for example here. There are also concerns on the part of students: "In hard economic times, students worry that professors who are stingy with the A’s will leave them at a disadvantage in graduate school admissions and employment." Interestingly, the NYT article states that: "Princeton adopted guidelines in 2004 providing that no more than 35 percent of undergraduate grades should be A’s, a policy that remains controversial on campus." According to the Ivy Gate Blog, Princeton have studied the effects on admissions rates to top medical schools and law schools, and found no effect. It was also reported in the NYT article that "Dartmouth transcripts include median grades, along with the number of courses in which the student exceeded, equaled or came in lower than those medians. Columbia transcripts show the percentage of students in the course who earned an A."

There is also reference to some academic research in the NYT article. In 1996, Cornell’s faculty adopted a “truth in grading” policy, and median grades were posted online starting in 1998. The policy called for median grades to be shown on transcripts as soon as student-records technology made that possible, but that did not happen until a full decade later. "While the median grades were available only online, a study by three Cornell economists found a large increase in enrollment in courses with a median grade of A — further driving grade inflation." This is suggestive of students enrolling in classes where it easier to obtain higher grade-scores; an issue which I raised before in relation to the Irish Leaving Certificate: the final examination at the end of second-level education Ireland.

The Lake Wobegon Effect in Student Self-Reported Data

The Lake Wobegon effect, as described by Wikipedia, is also known as illusory superiority or the above average effect. It is a "cognitive bias that causes people to overestimate their positive qualities and abilities and to underestimate their negative qualities, relative to others. This is evident in a variety of areas including intelligence, performance on tasks or tests and the possession of desirable characteristics or personality traits." The phraseology of the Lake Wobegon effect comes from Garrison Keillor's fictional town, Lake Wobegon, where "all the children are above average".

In 1994, Maxwell and Lopus published an article in the American Economic Review: The Lake Wobegon Effect in Student Self-Reported Data. According to the authors, overstated achievement may produce biased estimates of the relationship between achievement and educational inputs, if overstatement is correlated with achievement. Maxwell and Lopus produce evidence for biases which stem from two sources. First, below-average students tend to inflate their academic achievements, and second, they often fail to report their inferior accomplishments.

In a more recent paper (Journal of Economic Education, 2010), Haley, Johnson and McGee examine whether using student-survey data in place of official records data meaningfully biases regression estimates. They motivate their contribution by noting a useful statistical feature of overreporting on bounded variables such as grade point average. "Specifically, the misreports will be negatively correlated with the true grade point average, yielding a form of nonclassical measurement error that actually counteracts the bias." The authors connect this observation to reliability ratios used in labour economics. In two applications, they find that it is unnecessary to correct for the bias from the Lake Wobegon effect because it is so small.

Semesterisation's effect on exam results

"Trinity College will be releasing a report on semesterisation and its effect on students’ exam results next month, according to the college’s Communications Office." This is an excerpt from a story in The University Times, a Trinity college newspaper.

Lecture Attendance at Irish Universities

"If I’m Not Learning, Why Go?" This is the title of a blog-post from last year by Stephen Kinsella. Here's an excerpt: "One answer MIT surveyors found was when students didn’t feel they were learning, they didn’t go. This is certainly because the penalty to not going to a lecture is reduced by the presence of online learning materials like power point slides and handouts." Here's a link to a newsletter about the MIT Survey: Clay, T. and Breslow, L. (2006) “Why Students Don’t Attend Class“, MIT Faculty Newsletter, XVIII(4).

According to the MIT survey, students’ attitudes toward lectures vary widely, from "I never miss them" to "they’re worthless," with most responses falling somewhere in between. Most students reported they try to attend lectures, and usually do, missing them from time to time as the result of academic, extracurricular, or personal conflicts. "When asked to estimate what percentage of their lectures they attend, about two of every three respondents (67%) estimated that they attend at least 90%, three of every four (76%) that they attend at least 75%, and more than nine in 10 (93%) that they attend at least half."

Recent research by myself and others from Geary has shown that approximately 12% of Irish university students claim to attend all of their lectures. 32% of students claim to attend 90% or more of their lectures. 47% of students claim to attend 80% or more of their lectures. 57% of students claim to attend 70% or more of their lectures. Finally, 67% of students claim to attend at least half of their lectures. However, one must be cautious in drawing comparisons between the results from the MIT survey and the survey of Irish university students. The figures quoted above from the MIT survey are based on 47 responses by students in one subject. The figures quoted above from the Irish Universities Study are based on 2,867 responses across all subject areas. And of course, another (perennial) problem is social desirability bias: that is, students may say they attend their lectures because they think they should (say that).

According to Paul Latreille (H/T Stephen Kinsella), "falling lecture attendance among university students has become a major concern in the last few years. While the precise extent of the problem may vary, the phenomenon exists across subjects and has been documented in several countries (e.g. South Africa, Australia and the US, as well as the UK - see for example the references cited in Clearly-Holdforth, 2006). Various explanations have been advanced, including intrinsic factors (e.g. interest, motivation, learning styles and preferences), extrinsic factors (e.g. socio-economic considerations (such as the need to work), family commitments, assignment deadlines) and factors related to the lectures themselves (e.g. quality, value, interest)." Those interested in the role of technology in student learning will find an interesting discussion throughout Latreille's article. Laitrelle also covers the topic of "paternalistic" approaches to encouraging lecture attendance.

In 1993, David Romer published an article which ignited a lively debate about mandatory attendance policy: "Do Students Go to Class? Should They?" Daniel Marburger has investigated the impact of enforcing an attendance policy on absenteeism and student performance. The evidence he produces suggests that an enforced mandatory attendance policy significantly reduces absenteeism and improves exam performance. Alternatively, attendance at classes could also be potentially improved if students made a contribution toward the cost of their education. Finally, another possibility may simply be to communicate to students the importance of attending their lectures. Research papers by Schmidt (AER, 1983), Romer (JEP, 1993) and Durden and Ellis (AER, 1995) all show that lecture attendance is important for attaining high grades. Other studies from Irish settings also show the same: Maloney and Lally (1998), Kirby and McElroy (2003) and Purcell (2007).

Addendum: as is often the case in relation to other questions, establishing causality between lecture attendance and grades is difficult. Papers by Park and Kerr (1990), Bratti and Staffolani (2002) and Martins and Walker (2006) discuss this issue well. I am also reminded of the cartoon below.