Open Learning - Research Papers

UVA is taking advantage of the opportunity to conduct research and experiments in massive open online courses (MOOCs). Data from the MOOCs is at the intersection of big data science and learning analytics. Researchers are working to understand a variety of questions around learning in MOOC environments.

Recent research in the UVA MOOCs has focused on promoting peer to peer learning in discussion forums, using nudge email communications to influence retention, and providing methods for faculty to structure MOOC discussion forums to enhance academic relevance of learner discussion posts.


Recent papers include:

  • Communication Communities In MOOCs (Gillani, Eynon, Osborne, Hjorth, & Roberts, 20XX)

    The rise of Massive Open Online Courses (MOOCs) has brought together thousands of people from different geographies and demographic backgrounds, but to date, little is known about how they interact and learn. We introduce a new content-analysed MOOC dataset and use Bayesian Non-negative Matrix Factorization (BNMF) to extract communities of learners based on the nature of their online forum posts. We see that the communities BNMF learns are differentiated by their composite students' demographic and course performance indicators. We conclude with a discussion of how computationally efficient probabilistic generative models can be leveraged in conjunction with automated or crowd sourced content analysis to inform educators and learners about latent communication tendencies.

  • Communication Patterns In Massively Open Online Courses (Gillani & Eynon, 2014)

    Despite the hype and speculation about the role massively open online courses (MOOCs) may play in higher education, empirical research that explores the realities of interacting and learning in MOOCs is in its infancy. MOOCs have evolved from previous incarnations of online learning but are distinguished in their global reach and semi-synchronicity. Thus, it is important to understand the ways that learners from around the world interact in these settings. In this paper, we ask three questions: (1) What are the demographic characteristics of students that participate in MOOC discussion forums? (2) What are the discussion patterns that characterize their interactions? And (3) How does participation in discussion forums relate to students' final scores? Analysis of nearly 87,000 individuals from one MOOC reveals three key trends. First, forum participants tend to be young adults from the Western world. Secondly, these participants assemble and disperse as crowds, not communities, of learners. Finally, those that engage explicitly in the discussion forums are often higher-performing than those that do not, although the vast majority of forum participants receive “failing” marks. These findings have implications for the design and implementation of future MOOCs, and how they are conceptualised as part of higher education.

  • Experiences of Learning In a MOOC (Eynon, Hjorth, Gillani & Yasseri, 2014)

    To fully understand and support learners in MOOCs we need a way of detecting and differentiating distinct learner profiles (reflecting motivations, goals, preferences for course elements and online behaviours) - particularly given the “crowd like” nature of these settings. This paper highlights some steps towards this goal. Using a range of data sources from one case study MOOC on business strategy we address three questions: 1) can we distinguish coherent profiles of learners’ interactions within a MOOC? 2) how do these interaction profiles relate to learner characteristics? And 3) what is the relationship between these interaction profiles and learners’ performance and experiences of learning within a MOOC?

  • MOOCs As a Massive Research Laboratory: Opportunities and Challenges (Divera & Martinez, 2015)

    Massive open online courses (MOOCs) offer many opportunities for research into several topics related to pedagogical methods and student incentives. In the context of over 20 years of online learning research, we discuss lessons to be learned from observational comparisons and experiments on randomly chosen groups of students. We target two MOOCs for our study. We investigate dropout rates and how students who decide to drop out differ from those who continue courses. We discuss class forums and video lectures and how these interactions correlate with achievement. We explore the strong correlation between procrastination and achievement and implications for MOOC design. We examine the role of certifications offered by MOOCs and how different options can affect outcomes. We also examine the potential of linking data across courses. We discuss survey data in the context of these MOOCs. These research opportunities offer big data challenges, which are addressed with parallel computing techniques.

  • Never put off till tomorrow? (PDF) (Martinez, 2015)

    This paper identifies the causal effect of procrastination on achievement in a MOOC. I use two approaches: instrumental variables (IV) and a randomized control trial. I show that rain and snow affect when a student takes a quiz, and therefore can be use as an IV for procrastination. I fid that taking the course first quiz on the day it is published, rather than procrastinating, increases the probability of course completion by 15.4 percentage points. With the randomized control trial, I show that very low-cost intervention can increase student achievement. I send an email (directive nudge) encouraging a randomly selected group of students to procrastinate less. Students assigned to the treatment group were 16.85% more likely to complete the course. I also find that the effects are heterogeneous across countries, suggesting that it may be advisable to customize nudges to country characteristics. This online experiment may also provide valuable lessons for traditional classrooms.

  • Structural Limitations Of Learning In a Crowd (Martinez, 2015)

    Massive Open Online Courses (MOOCs) bring together a global crowd of thousands of learners for several weeks or months. In theory, the openness and scale of MOOCs can promote iterative dialogue that facilitates group cognition and knowledge construction. Using data from two successive instances of a popular business strategy MOOC, we filter observed communication patterns to arrive at the “significant” interaction networks between learners and use complex network analysis to explore the vulnerability and information diffusion potential of the discussion forums. We find that different discussion topics and pedagogical practices promote varying levels of 1) “significant” peer-to-peer engagement, 2) participant inclusiveness in dialogue, and ultimately, 3) modularity, which impacts information diffusion to prevent a truly “global” exchange of knowledge and learning. These results indicate the structural limitations of large-scale crowd-based learning and highlight the different ways that learners in MOOCs leverage, and learn within, social contexts. We conclude by exploring how these insights may inspire new developments in online education.

  • The Hawthorne Effect in MOOCs - coming soon
  • The Effects of Informational Nudges On Students' Effort and Performance: Lessons From a MOOC (Martinez, 2015)

    I evaluate the impact of providing students with information about their performance relative to their classmates. I run a randomized experiment in the context of a Coursera Massive Open Online Course (MOOC), assigning students to either one of two potential treatments. The first, framed positively, describes what fraction of his classmates a student outperformed. The second,framed negatively, describes what fraction of his classmates a student under- performed. I find evidence that students respond to this informational nudge and that framing matters. Students who were doing relatively poorly respond to the negative treatment with more effort, and this effort translates, in some cases, into higher achievement.

  • Uncovering Latent Features In Massively Open Online Courses (Gillani & Michaelmas, 2014)

    Massive Open Online Courses (MOOCs) bring together a global crowd of thousands of learners for several weeks or months. We use social network analysis and community detection to uncover the latent features of online discussions in MOOCs. We begin by using data from two successive instances of a popular business strategy MOOC to filter observed communication patterns and arrive at the "significant" interaction networks between learners. We then use complex network analysis to explore the vulnerability and information diffusion potential of the discussion forums. While network analysis offers a vibrant post-hoc analytical framework, it fails to answer a fundamental question: can we devise a model to represent the generation of the data set at hand?

    Moving from the structural properties of global-scale discussion to the discussion content itself, we employ existing educational theories to qualitatively content-analyse over 6,500 forum posts from a particular MOOC. We then use a generative model - Bayesian Non-negative Matrix Factorization (BNMF) - to extract communities of learners based on the nature of their online forum posts. We observe that the inferred communities are differentiated by the nature and topic of dialogue, as well as their composite students' demographic and course performance indicators. While qualitative analysis confirms these detected communities, additional quantitative sensitivity analysis shows that they are not crisply defined, illuminating key challenges of applying Machine Learning techniques to model noisy and incomplete learner data.

    We conclude by discussing the key insights of this work for online education, namely, that different discussion topics and pedagogical practices promote varying levels of peer-to-peer engagement. Additional qualitative investigations reveal that many learners feel a sense of "content-overload" when deciding to participate in online discussions, often leading to their disengagement. These insights call for an interdisciplinary effort to help create relevant and personalized learning experiences in massive scale online settings.

  • Learner Communications In Massively Open Online Courses (Gillani, 2013)

    Until now, most have not offered formal institutional credit but have been freely available to anyone with an internet connection, regardless of their educational background. MOOCs have become popular topic in higher education largely because they enable a geographically diverse group of learners to access educational resources from the world’s top universities. They have evolved from previous incarnations of online learning but are distinguished in their global reach and semi-synchronicity.

    In the past two years, MOOCs have received very polarized media attention. Some believe MOOCs will completely transform traditional models of higher education. Others view them as mechanisms for furthering a commoditization of learning that is best experienced in small groups and in-person. Unfortunately,a great deal of this debate has lacked theoretical grounding and evidence from rigorous research. Sound investigation is needed to move beyond these extreme views and evaluate the true pedagogical potential of MOOCs.

    This work analyses a key differentiator of MOOCs from previous efforts at open education–communication between a global body of learners via online discussion forums – to discover who tends to interact online and how. Literature on MOOCs has not yet indicated the backgrounds, motivations, or achievement levels of forum participants. It also has not revealed their communication patterns or how groups tend to form and disband around certain topics. As MOOCs enable communication between learners that may have otherwise never interacted, it is essential to gain insights into how they engage in online discussions to better support their learning.

    This study aims to address this need. Data analysis of nearly 87,000 individuals from a case study of a particular MOOC reveals a number of key trends. Forum participant/– like those in the course more broadly – tend to be young adults from the western world. Students in the course favour “real-world” topics that have relevance and significance in their lives beyond the academic setting. Forum participants assemble and disperse quickly as crowds, not communities, of learners. Finally, those that engage explicitly in the discussion forums are often higher - performing than their counterparts in the course, although the vast majority of forum participants receive “failing” marks. These findings have implications for how certain types of MOOCs may encourage and promote online discussions in the future – and how these discussions can help students learn.