In Papers and Talk, Researchers Say EdX Success Not Tied to Dropout Rate

One of the lead authors of a series of working papers analyzing data from the first full academic year of edX courses said Tuesday night that the high dropout rate of course registrants on the platform is not an adequate metric to measure the program’s success.

That was after the papers, released earlier Tuesday, showed that 95 percent of registrants for courses offered by Harvard and MIT on edX, the open online education platform co-founded by the two universities, did not earn certificates of completion—a statistic that made headlines across the country.

In his discussion of the findings at Tuesday evening’s event at the Graduate School of Education, HarvardX Research Fellow Justin Reich maintained that these course certification rates on their own could not accurately portray the extent to which users are learning on the platform and that researchers are still looking for more informative metrics to evaluate online learning.

“A certification rate’s a funny thing,” Reich said. “The inferences that we are making based on these rates are probably not well supported. There are other ways that we could be thinking about course quality.”

In particular, Reich emphasized that classroom pedagogy does not provide a gold standard with which to evaluate online education because the two formats are different both in terms of their structure and expectations.

“The ways in which you interpret any of these data depends quite a bit on your frame of reference. I would encourage you, as you read through the reports, to play around with your frames of reference,” Reich said to a filled lecture hall of mostly graduate students and faculty members.

For instance, using the date of enrollment as a point of reference, the researchers found that the high drop rates fell substantially within a week or two after enrollment. After this initial window, the fraction of users who drop out decreases significantly, according to the reports, which analyzed data compiled from the 17 courses offered by Harvard and MIT during the 2012-2013 academic year.

Reich was one of several speakers at the event, entitled “The First Year of HarvardX: Research Findings to Inform the Future of Online Learning,” which was co-sponsored by HarvardX, the GSE, and the Berkman Center for Internet and Society. He was introduced by Vice Provost for Advances in Learning Peter K. Bol, who oversees HarvardX.

Andrew D. Ho, an associate professor of education and co-chair of the HarvardX research committee, said in an interview before the event that the small percentage of registrants who earn certification actually translate to large real numbers, given edX’s large scale. With a total of 841,687 registrations, the 5 percent figure means that 43,196 registrants earned certificates.

In addition to enrollment and certification numbers, the large sets of data—averaging at 20 gigabytes per course—provided new insights into the current demographics of MOOC users, including age, gender, country, and previous education.

“[The reports] provide a nuanced description of the variation across open online courses and our hope is that they advance the discourse around these surprisingly rich and rapidly advancing learning ecosystems,” Ho said. “They show in great detail the demographic profiles of our registrants. Now that we know those, we can work to change those profiles through selective recruitment to meet our pedagogical goals.”

Ho attributed the comprehensiveness of the research to the ongoing collaboration between Harvard and MIT on the edX platform.

While giving summary of the researchers’ findings, the papers do not make public the raw data pertaining to the courses. Data sets and “interactive visualization tools” will be be released in the coming weeks, according to an edX press release.

Meanwhile, HarvardX will launch 14 new courses in the spring 2014 semester, with classes covering topics including economics, genomics, and history.

Toward the conclusion of his presentation, Reich mentioned that further educational experimentation would be incorporated into future edX courses as part of the organization’s continuing efforts to improve the accessibility and quality of education.

—Staff writer Michael V. Rothberg can be reached at mrothberg@college.harvard.edu.