Stats Experts Point to Flaws in UC Social Group Survey

The Undergraduate Council’s recent survey on single-gender social organizations possessed a number of methodological flaws, according to survey design experts, lending weight to criticism that emerged in the wake of the survey’s release on Wednesday.

Results from the survey, which UC leaders sent to University President Drew G. Faust last week, indicated that 61 percent of respondents opposed penalties on social groups. Yet only 28 percent of undergraduates participated in the survey, leading some students and experts to question the study’s validity.

Chase H. Harrison, the Associate Director of the Harvard Program on Survey Research, pointed out that the survey may have suffered from non-response bias, a phenomenon in which those with strong feelings about the survey’s subject are more likely to respond than those without strong feelings on the subject.

“Without some attempt to measure who responded, adjust who responded, it’s difficult to know what to make of it,” Harrison said.

Harvard’s current policy, which took effect for the Class of 2021, bars members of unrecognized single-gender social organizations from holding athletic team captaincies, becoming leaders of various campus groups, and receiving College endorsements for prestigious fellowships.

Government Professor Gary King, who directs the Institute of Quantitative Social Science, said that without a survey question about club affiliations, it is difficult to tell whether or not members of single-gender social organizations disproportionately responded to the survey.

“They didn’t collect the information, so you can’t even know. And that’s the problem,” he said.

UC President Yasmin Z. Sachee ’18 and UC Vice President Cameron K. Khansarinia ’18 sent a link to the survey to undergraduates on Oct. 26, and closed the survey eight days later. The survey did not prompt respondents for their social group affiliation.

“We weren’t able to do a randomized, selective survey sort of thing, for a number of reasons—one of which is a lot of the lists of students in these organizations are not public, so we wouldn’t have really known where to start there,” Sachee, who is a member of the Bee Club, said.

Donald B. Rubin, a professor of statistics, pointed out the importance of effective follow-up, which he suggested was lacking in the UC survey.

“There are other kinds of adjustment that you can do that is based on how long it takes for somebody to respond,” he said. “Some people will respond at the next attempt.”

UC President Yasmin Z. Sachee '18 and UC Vice President Cameron K. Khansarinia '18 . By Soumyaa Mazumder

Amid criticism, Khansarinia downplayed the importance of the quantitative data from the survey.

“I think if people had taken the time to read the five-page report in its entirety, they would have known that we have already discussed these things at length—they would have known that we took so much value from the long comments and didn’t place a huge amount of value on, you know, the shock value numbers of 60 percent say this, 20 percent say this, because we knew the limitations of the survey,” he said in an interview.

Although the UC report prefaces its findings with an acknowledgment that the survey was “imperfect in many ways,” it goes on to state that “proposed alternatives” to the sanctions are “still unpopular.” The report decries an option proposing a full ban on social groups, pointing to the survey as evidence of its unpopularity.

“Following our study of this issue, we firmly believe that a full ban of social groups would violate students’ right to choice and would be ineffective,” the report said.

The report calls on Faust to choose a policy solution between a total ban and the status quo. Faust said in an interview earlier this month that she will make a final decision after the next Faculty meeting in December.