MOOC success factors: Proposal of an analysis framework

Volume 16, 2017

Accepted by Editor Peter Blakey│ Received: May 23, 2016│ Revised: July 24, September 5, 2017 │ Accepted: September 7, 2017.

Cite as: Azevedo, J., & Marques, M. M. (2017). MOOC success factors: Proposal of an analysis framework. Journal of Information Technology Education: Innovations in Practice, 16, 233-251. Retrieved from

http://www.informingscience.org/Publications/3861

(CC BY-NC 4.0) This article is licensed to you under a Creative Commons Attribution-NonCommercial 4.0 International

License. When you copy and redistribute this paper in full or in part, you need to provide proper attribution to it to ensure

that others can later locate this work (and to ensure that others do not accuse you of plagiarism). You may (and we encour-age you to) adapt, remix, transform, and build upon the material for any non-commercial purposes. This license does not permit you to use this material for commercial purposes.

MOOC

S

UCCESS

F

ACTORS

:

P

ROPOSAL OF AN

A

NALYSIS

F

RAMEWORK

José Azevedo* University of Porto, Porto, Portugal [email protected] Margarida Morais Marques CIDTFF, University of Aveiro,

Aveiro, Portugal [email protected]

* Corresponding author

A

BSTRACT

Aim/Purpose From an idea of lifelong-learning-for-all to a phenomenon affecting higher edu-cation, Massive Open Online Courses (MOOCs) can be the next step to a truly universal education. Indeed, MOOC enrolment rates can be astoundingly high; still, their completion rates are frequently disappointingly low. Nevertheless, as courses, the participants’ enrolment and learning within the MOOCs must be considered when assessing their success. In this paper, the authors’ aim is to re-flect on what makes a MOOC successful to propose an analysis framework of MOOC success factors.

Background A literature review was conducted to identify reported MOOC success factors and to propose an analysis framework.

Methodology This literature-based framework was tested against data of a specific MOOC and refined, within a qualitative interpretivist methodology. The data were collected from the ‘As alterações climáticas nos média escolares - Clima@EduMedia’ course, which was developed by the project Clima@EduMedia and was submitted to content analysis. This MOOC aimed to support science and school media teachers in the use of media to teach climate change.

Contribution By proposing a MOOC success factors framework the authors are attempting to contribute to fill in a literature gap regarding what concerns criteria to consider a specific MOOC successful.

Findings This work major finding is a literature-based and empirically refined MOOC suc-cess factors analysis framework.

Recommendations

for Practitioners The proposed framework is also a set of best practices relevant to MOOC devel-opers, particularly when targeting teachers as potential participants. Recommendation

MOOC Success Factors

234

Impact on Society By providing a proposal of a framework on success factors for MOOCs, the au-thors hope to contribute to the quality of MOOCs.

Future Research Future work should refine further the proposed framework, by testing it against data collected in other MOOCs.

Keywords distance education, lifelong learning, Massive Open Online Courses (MOOCs), success factors, framework, content analysis, climate change

I

NTRODUCTION

The term Massive Open Online Course (MOOC) was coined for the 2008 edition of the course ‘Connectivism and Connective Knowledge’ (Kady & Vadeboncoeur, 2013). In less than a decade, the MOOC movement evolved from an idea of lifelong-learning-for-all to a phenomenon affecting higher education on a global scale (Walker & Loch, 2014). High-profile universities, such as Stanford and Harvard, were among the early providers, introducing extra attention and media coverage (Bates, 2014; Kovanović, Joksimović, Gašević, Siemens, & Hatala, 2015) and, thus, created an unprecedented public interest. Nowadays, MOOCs can be described as articulated sets of learning activities and resources, web-based, usually free-of-charge and with no prerequisites, which can be accessed simul-taneously by hundreds of users.

Several advantages of MOOCs are pointed out in the literature, as they can:

• provide high quality, low cost and high scale education (Saadatdoost, Sim, Jafarkarimi, & Mei Hee, 2015; St Clair, Winer, Finkelstein, Fuentes-Steeves, & Wald, 2015);

• _{increase the access to higher-education learning (St Clair et al., 2015; Walker & Loch, 2014);} • _{promote autonomous, independent and flexible learning (Kady & Vadeboncoeur, 2013;}

Saadatdoost et al., 2015);

• allow learners to focus on learning rather than on getting a qualification (Walker & Loch, 2014); and

• _{promote the provider - institution, professor or study programme (Saadatdoost et al., 2015;} St Clair et al., 2015; Walker & Loch, 2014).

Hence, MOOCs seem to have the potential for a truly universal education as they allow delivering high-quality learning content to individuals that usually cannot access higher education (Bates, 2014). However, the typical course registrant of, for example, Harvard and MIT Open Online Courses is a well-educated young male from a developed country (Ho et al., 2015; Ho et al., 2014). Hence, one might argue that MOOCs have not delivered yet the promise of providing education to the individu-als who could benefit the most from their openness and free features (Christensen et al., 2013). Addi-tionally, according to a study of 221 MOOCs, completion rates vary from 0.7% to 52.1%, with a disappointing median value of 12.6% (Jordan, 2015). Kovanović et al. (2015) claim that low comple-tion rates created the need of understanding the factors that drive students’ success in MOOCs. Other authors, however, are questioning if the high dropout ratio should be considered an issue in this context (Ho et al., 2015; Saadatdoost et al., 2015) and, hence, other dimensions need to be taken into consideration.

S

P

,

P

M

’

O

From the discussion above, there is insufficient literature reporting factors that influence MOOCs’ success and further research is needed to fully understand what makes a specific MOOC successful. Aiming to contribute to the literature, the authors sought to identify a set of success factors that may affect the enrolment, continuance, and learning of MOOC participants, in order to propose a MOOC success factors framework. This framework can be used both to support MOOC providers in the development of their courses and in the analysis of their MOOC’s success. With that aim, firstly, a literature review of MOOC success factors is presented and discussed (in the following

sec-Azevedo & Marques

235 tion of this paper), in order to propose an initial framework. Secondly, in a qualitative interpretivist approach, this framework was used in a content analysis of data collected from the MOOC ‘As al-terações climáticas nos média escolares - Clima@EduMedia’ [Climate change in the school media - Clima@EduMedia]. The context of this MOOC, the data collection and analysis techniques, as well as the limitations of this study are described in the ‘Methodology’ section. The analysis allowed us to test and refine the framework, which is of particular importance since we face a scarcity of empirical research on the MOOC phenomenon (Kady & Vadeboncoeur, 2013), particularly considering the participants’ perspectives (Gamage, Fernando, & Perera, 2015). The results of this analysis are pre-sented and discussed in the presentation and discussion of the results’ section. Finally, in the ‘con-cluding remarks’ section, a reflection of what success factors were more relevant for the participants of the selected MOOC is also presented.

The authors consider that the literature-based and empirically refined MOOC success factors frame-work, presented in Table 1 (in the ‘Methodology’ section), is an important contribution of this work and can be used as a guide for best practices for MOOC providers. This framework is particularly relevant in the field of teacher development, as this was the context of this study.

L

ITERATURE

R

EVIEW

MOOCs were born from connectivism, which was proposed by George Siemens in 2005 as a learn-ing theory of the digital era. The author argued connectivism overcomes limitations of three of the main broad learning theories – behaviourism, cognitivism, and constructivism – namely, not consid-ering the current fast development of technology and knowledge-base, as well as learning as a phe-nomenon that can occur outside people and within organizations. In his first published reflection on this theory, Siemens highlights the increasing need to assess the learning worthwhile of pursuing, in a society where access to information, and to knowledge, increases exponentially. Not being able to learn all, people need to filter information and learn how to access the information they need. Thus, in connectivism the focus is on the connections that enable the individual to learn, rather than his or her current state of knowing (Siemens, 2005). Despite an increasing acceptance of this learning theo-ry, connectivism has not been widely accepted and some critical voices have emerged. Some authors argue it is not a learning theory, but rather a curriculum theory. Others claim that connectivism (i) cannot explain a range of learning phenomena, (ii) is void of new ideas, (iii) presents an ill-defined teacher role, and (iv) requires from the learner a high motivation level for autonomous en-gagement with the resources and for maintaining interaction with others MOOC participants (An-derson, 2016).

A consequence of the above is that connectivism has not been adopted by all MOOC providers and this mode of education delivery soon evolved into different configurations. In fact, MOOC literature frequently distinguishes between cMOOCs and xMOOCs. cMOOCs are associated with connectiv-ism, as learners are asked to contribute to the course’s development by discussing topics and bringing in various forms of relevant content (Admiraal, Huisman, & Pilli, 2015; Kennedy, 2014; Saadatdoost et al., 2015; St Clair et al., 2015) and frequently use distributed online platforms (Kennedy, 2014). These courses have a social approach to education (Liyanagunawardena, Adams, & Williams, 2013) and require innovative ways of teaching (Saadatdoost et al., 2015). xMOOCs are highly structured and instructor guided, usually including a set of video lectures, recommended readings, and different forms of assignments and assessments in a centralized platform (Kennedy, 2014). These courses have an individualist learning approach (Liyanagunawardena et al., 2013) and are grounded in cogni-tive-behaviourism (Stacey, 2014). Alternative terms and classification systems have been proposed, due to the high diversity of MOOC designs and purposes (Admiraal et al., 2015; Drake, O'Hara, & Seeman, 2015), although their usefulness can be rather limited.

It is fairly acknowledged that different stakeholders have different motivations regarding MOOCs, making success and quality assessment challenging. For example, a MOOC success can be linked to creating a good image of the MOOC promoter (Walker & Loch, 2014) or to each learner meeting his

MOOC Success Factors

236

or her own goals (Downes, 2016). In this last example, pass rates are not be a reasonable success indicator for a MOOC, as course completion may not be the enrollee’s major goal (e.g., Alraimi, Zo, & Ciganek, 2015; Rosewell & Jansen, 2014). Yet, the question remains: what elements should be con-sidered when analysing a MOOC success?

Currently, several authors are discussing MOOC quality-related topics, including Downes (2016), Yang, Shao, Liu, and Liu (2017), and Creelman, Ehlers, and Ossiannilsson (2014). Not many are dis-cussing success factors (some exceptions are Alraimi et al., 2015, Nagashima, 2014, and Poy & Gon-zales-Aguilar, 2014); hence, the need for research in this area.

The MOOC success factors framework presented in this paper (in Table 1) was organised according to Nagashima’s (2014) proposal of three sets of categories of success factors for open education initiatives, including MOOC aggregators: organisational, pedagogical, and social. Each of these sets includes several success factors identified either in the literature, in the data collected or in both. Be-low, the authors discuss the literature-based MOOC success factors.

L

-

MOOC

S

F

Downes (2016) stresses the relevance of identifying what a successful MOOC ought to produce as an output, particularly, participants’ autonomy, allowing diversity, openness, and interactivity. ‘Auton-omy’ is essential for participants being able to pursuit their own goals (and not necessarily the ones pointed by the MOOC providers). ‘Diversity’ refers to the different approaches MOOC participants have when engaging with MOOC’s activities. This factor allows the MOOC to be relevant for people with different cultures, time zones, available technologies, learning styles, and other distinctive charac-teristics. ‘Openness’ highlights the free flow of people and of information in the MOOC, that is, people are free to join the MOOC, leave it, access produced content, and bring in their own re-sources. Finally, ‘interactivity’ allows the emergence of new learning within the network of partici-pants of the MOOC (MOOC tutors and providers included). Hence, these three aspects contributed to the definition of the categories ‘Flexibility and Scaffolding for Diversity’, ‘Openness’, and ‘Interac-tivity and Peer-to-Peer Pedagogy’, respectively, all included in the framework summarised in Table 1. Rooij and Zirkle (2016) and Nagashima (2014) also refer to interactivity, that is, the connections among all MOOC participants, as an enhancer of student engagement. Furthermore, this factor’s effect can be increased by knowing other learners in the MOOC, who can promote engagement with the course resources (Kizilcec & Schneider, 2015). Therefore, ‘Acquaintances’ was another success factor presented in Table 1.

Nagashima (2014) also acknowledges other success factors, such as:

• the MOOC’s funding strategies, as the MOOCs are frequently offered without charging fre-quency fees, this is a relevant factor for MOOC providers. This led to the success factor ‘Funding’ (Table 1);

• localisation, to get around disparities in technological infrastructure, language, and, even, cul-ture of MOOC participants, leading to the success factor with the same designation in Table 1;

• focus of subjects, a factor driven by the learners’ reasons for taking these courses, as their motivations for enrolment might be different for different subject areas. This factor originat-ed the success factor with the same name in Table 1;

• social view of open education, that is, the way society responds, recognizes, and accepts the idea of open education. This factor originated the factor ‘Learning View’ in Table 1, after testing the literature-based framework against this study’s empirical data. This change was due to the fact that the user’s view of open education as having, or not, high quality can be related with his or hers perspective regarding learning and the strategies to achieve it.

Azevedo & Marques

237 The provider’s reputation, as perceived by the user, is a factor worth taking into account as it may influence the decision of enrolling in a specific MOOC (Alraimi et al., 2015); hence, the inclusion of the success factor ‘Reputation or Brand’ in Table 1. Moreover, according to Yang et al. (2017), the learners’ decision of continuance in a MOOC seems to be influenced by several factors, such as:

• _{system quality, that is, the functionality and reliability of the MOOC’s supporting} technolo-gies;

• course quality, defined by the knowledgeability, authority of course content, and lecturers’ teaching attitudes;

• _{service quality, or the support from the MOOC’s provider or other MOOC learners, e.g., to} help users to engage in learning tasks;

• perceived ease of use of the online system; and

• perceived usefulness of the online system in achieving own goals.

These factors, proposed by Yang et al. (2017), contributed to originate several success factors in Ta-ble 1: ‘Technology’, ‘Quality Resources’ and, again, ‘Interactivity and Peer-to-Peer Pedagogy’. Kizilcec and Schneider (2015) found that the intention to earn a certificate promotes students’ as-signment attempt. Therefore, offering learning recognition seems to be a strong predictor of user participation and continuance in the MOOC, and it was included in the framework as the success factor ‘Credits Recognition’.

In the quality discussion thread, under the MOOC quality project, Creelman et al. (2014) propose some key areas related to the perception of MOOC quality, such as providing clear pre-course in-formation, such as the course’s structure or the expected workload, to set adequate expectations (originated the factor with the same name in Table 1) and mixing formal (for credits) and informal learners (for self-development). In the same context, Rosewell and Jansen (2014) distinguish eight relevant key principles suited to MOOCs, namely, openness to learners (related with the free admis-sion of participants and to different ways of participation) and media-supported interaction (refer-ring to the use of rich media, namely video and audio, and to the interactivity the online medium supports). Walker and Loch’s (2014) study pointed out some features that can contribute to MOOC quality in the perspective of academics as MOOC users. They mentioned the need to design the course accounting for its potential massive enrolments (hence, the factor ‘Scale or Massiveness’ in Table 1) and the importance of effective feedback regarding the participants’ learning (included in the factor ‘Interactivity and Peer-to-Peer Pedagogy’, mentioned before). In a pedagogical focused perspective, Stacey (2014) presented five recommendations, including (i) promoting peer-to-peer pedagogies, where participants can provide feedback or assessment comments to other participants; if enough support is given, this peer-to-peer interaction can allow MOOC sustainability even at a large scale; (ii) openness of course resources, leveraging the entire web instead of just the content in the MOOC platform and publishing its resources under Creative Commons licences, and (iii) open-ness to enrolments.

In a more comprehensive approach, Gamage et al. (2015, p.227) sought “to identify the factors affect to quality of MOOC.” From their literature review, they argue that factors with a significant role in making a MOOC effective to a learner are (i) interactivity (with resources, instructor and peers), (ii) collaborativeness, which allows learning through social interactions; (iii) pedagogy (namely learning pace), and (iv) technology (particularly the support regarding hardware and software). On the other hand, there are challenges that need to be adequately addressed, such as information overload and its relation to MOOC participation. All these factors were integrated in the framework summarised in Table 1.

More recently, Yepes-Baldó et al. (2016) present a 14 dimensions evaluation system of MOOC quali-ty that included, for example, a proper selection and organization of the MOOC’s contents and a temporary configuration allowing customisable schedules and autonomous rhythms. These factors

MOOC Success Factors

238

contributed to include the success factors ‘Content Organisation and Access’ and ‘Timing’, respec-tively, in Table 1.

In sum, based on this literature review and departing from Nagashima’s (2014) general categories of success factors, we developed an analysis framework of MOOC success factors, which is presented bellow in Table 1 (first two columns). Other literature (DeBoer, Ho, Stump, & Breslow, 2014; Drake et al., 2015; Hernández, Morales, Mota, & Teixeira, 2014; Kennedy, 2014) not directly on success factors or quality of MOOCs was considered, as their findings seemed to support some items of our literature-based analysis framework. As mentioned in the previous section, the model was used to analyse data from a specific MOOC, as will be explained hereinafter.

M

ETHODOLOGY

The aim of this study was to propose a MOOC success factors framework, as the literature has not provided, so far, such a proposal. For this initial proposal, and as both numerical and non-numerical data from a specific MOOC were viewed as symbolic representations of the studied phenomenon requiring interpretation, a qualitative interpretivist approach was conducted (Twining, Heller, Nuss-baum, & Tsai, 2017). This approach allowed identifying and empirically validating a set of factors that may affect the enrolment, continuance, and learning of MOOC participants, an unusual but needed approach in MOOC research (Gamage et al., 2015). The framework proposed in this study, as will be discussed further, should be refined in future work.

C

–

MOOC

‘A

ESCOLARES

-

C

@E

M

’

We used data from the first MOOC provided by the University of Porto (Portugal). This Portuguese language MOOC was offered through the online MOOC platform miríada x. The main aim was to support science and school media teachers in the development of the skills needed to use media to teach climate change. Although this MOOC was open to anyone interested, both its structure and strategy of information diffusion to promote enrolments attracted mainly teachers, especially of the 3rd cycle and secondary levels of the Portuguese education system. The course’s goals included (i)

disseminating a set of documents with proposals of teaching strategies and (ii) promoting ideas and experiences sharing among the participants. The choice of this MOOC was based on the access to the data, as this paper’s authors were part of its teaching team.

The course ran from October 5th to November 14th, 2015. It had five content modules, with a weekly

staggered release. Online resources included content-videos, extension manuals, teaching strategies, discussion forum, peer assessment, and quizzes. Despite having a xMOOC configuration, some ef-forts were made to promote learners’ participation in the discussion of contents, and, hence, allowing some connectivism.

Free accreditation, relevant for Portuguese teachers, was also offered. To earn the accreditation, teachers were asked to attend a face-to-face session and writing a report about their MOOC experi-ence. A hundred and fifty-nine teachers presented a report for accreditation and, of these, 83.5% stated this was the first time they participated in a MOOC.

With 720 enrolments, 551 users who logged in at least once, and 311 participants that finished all the mandatory activities, this MOOC completion rate was 43.19%, surpassing the literature reported median value (Jordan, 2015).

D

C

A

As mentioned above, the teachers who aimed to get a professional development accreditation for the MOOC completion were asked to submit a written report through the online platform. The collec-tion instrument was an open answer response box, although teachers were asked to describe their

Azevedo & Marques

239 motivation and expectations for the course, their appreciation of the work, as well as their learning and its usefulness for their teaching practice. These topics prompted teachers to mention factors influencing their enrolment, continuance, and learning within the MOOC. Gathering data from re-sponses to open-ended questions is a frequent source of information in content analysis (Twining et al., 2017).

Each written report functioned as a case. The 159 reports presented by the teachers formed our data. The collected data were submitted to content analysis (Krippendorff, 2004a), supported by the soft-ware SPSS to compute descriptive statistics and determine inter-rater reliability (Krippendorff, 2004a). The aim was to check if the data supported the relevance of the literature-inspired analysis framework, presented in Table 1. The success factors (underlined text in the table) were grouped into three categories (in bold) inspired in Nagashima’s (2014) work. For each success factor a brief de-scription, support from the literature, support from the data, or both, are presented. The exceptions are ‘Funding’, with no data occurrences, and ‘New experience view’, which was included in the sis framework due to its recurrent appearance in the reports. The empirical contribution to the analy-sis framework is marked with text in italics in Table 1.

For the analysis, the authors assumed that each case could contain several units for coding, as each teacher could make reference to more than one success factor. Success factors were defined as fac-tors influencing users’ enrolment, participation, or both in the MOOC. Being a more subjective con-cept than enrolment, participation was defined as taking part in activities, such as accessing and ana-lysing the resources, reading and posting in the forum, or attempting quizzes.

The units for coding were defined as segments of text with the information needed for the analysis (Krippendorff, 2004a), in this study, information about success factors. Noncontiguous segments of text were considered in the analysis (Krippendorff, 2004a), as shows the following excerpt: “The motivations for the frequency of this course were: 1. …; 3. deepening of the scientific and pedagogi-cal approach to the theme «Climate Change» through the use of the media” (Report 21).

Unreliable or ambiguous information was not considered, due to difficulties in including it in only one category (Krippendorff, 2004a). For example, in the quote “As aspects for improvement, I would say that it was not possible for me to see all the videos in a timely fashion” (Report 2), the teacher did not explicitly present the barrier to video watching (it could be time constraints, technological diffi-culties or other). Text segments without information about success factors were also not considered in the analysis. For example, segments with information about the participant’s identity, as illustrates the quote “Name: [name removed for anonymity]; Subject matter: Biology and Geology; School: [name removed for anonymity]” (Report 11), or general statements or reflections about education, such as “[We aim], to form citizens more democrat and tolerant to difference” (Krippendorff, 2004a).

The reports were coded according to the framework presented in Table 1. Only one code, or success factor, was attributed to each coding unit. A ‘+1’, ‘0’ or ‘-1’ was used when the reported success fac-tor promoted, did not influence or inhibited the enrolment, the participation, or both, in the MOOC, respectively. However, the code ‘2’ was used when it was found contradictory information regarding the influence of a specific factor in a report. For example, the Report 74 was coded with ‘2’ in the ‘Technology’ category: “The platform, really well conceived, allows an efficient and fruitful use. It promotes interactivity and cooperation” and “As points for improvement, the platform could have had a way to better organise the forum contributions.” Hence, success factors were considered nom-inal variables.

Ta bl e 1 . M O O C s uc ce ss fa cto rs fr am ew or k a nt i ts suppo rt fro m the li te ra tu re a nd f ro m th e e m pi ric al d at a. SE T OF CA T EG O R IES OF SU CC E SS FA CT OR S AN AL YSI S CA T E G OR IE S O R SU CC E SS FA CT O R S SU PP O R T F R O M T H E L IT E R AT U R E SU PP O R T F R O M T H E E M PI R ICA L D AT A Soc ia l F ac tor s: re lat ed w ith the w ay soc iet y ca n infl ue nc e a M O O C’ s su c-ce ss Lea rn in g V iew : I nc lu de s r ef er enc es to 1) ap pr eci ati on of le ar nin g o ppo rtu nit ies ou tsi de of fo rm al edu cat ion ; 2 ) a ck no w led gm en t o f o pen ed uca tio n as p ot en tia lly ha vi ng h ig h qu ali ty e.g ., ‘h ow soc iet y re spond s, re cog ni ze s, and a cc ept s t he id ea of ope n ed uc at io n infl ue nc es th e ev ol ut io n of th e op en ed uca tio n m ov em en t.’ (N ag ash im a, 20 14 , p .1 7-18) e.g ., ‘p ro fe ss io na l d ev el op m en t, no t o ut o f o bl ig at io n, but b y t he sh ee r d es ire to le ar n, m us t b e p ar t o f a te ac he r-of -e xc ell enc e’s pa th. T he re for e, … I ha ve en ro lle d’ (R ep or t 126) Rep ut at io n or B ra nd : I nc lu de s r efe re nc es to 1) a h ig h rep ut at io n of th e pr ov id er (w hi ch ca n inc re as e pot ent ial u se rs ’ a w ar en es s of the M O O C and it s pe rc eiv ed q ua lit y) e.g ., ‘a br an d or rep ut at io n m ak es in iti at iv es w id ely sp re ad , co ver ed by m ed ia, an d kn ow n by peo pl e in th e w or ld … a n in st itu tio n’ s br and a lso affe ct s t he pe rc eiv ed q ua lit y of co nt ent s i t of fe rs .’ (N a-ga sh im a, 2014, p .16) ; O th er a ut ho rs : A lra im i e t a l. (2015, p .33) , D ra ke e t a l. (201 5) e.g ., ‘T he po ss ib ili ty of le ar ni ng a nd d ev elo pi ng sk ill s in th is fiel d w ith in a p res tig io us in st itu tio n ad ded to m y hi gh ex pect at io ns ’ ( Rep or t 7 2) Lo cal isat io n: Inc lu de s r efe re nc es to 1) d iffe re nt us er s’ fe at ur es d ue to the ir di ve rs e w or ld loc a-tio ns , s uc h as d iff er en t a cc es s t o te ch no lo gy, lan gu ag e skill s, tim e zon es , c ul tu re s, et c. e.g ., ‘T o get a ro un d di sp ar iti es in tech no lo gi ca l i nf ra st ru ct ur e an d di ffe re nc es in lang ua ge , s om e ini tia tiv es ha ve a tte m pt ed to loc ali ze th eir c ou rse s a nd h av e ga in ed p op ul ar ity .’ (N ag ash im a, 20 14 , p .1 8) ; O th er a ut ho rs : D ow ne s ( 2016 , p ar a. 69) , D ra ke e t a l. (201 5, p .132) , K en ne dy (2014, p .8) , K iz ilc ec & S ch ne id er (2015, p .6: 19) , R os ew ell & Ja ns en (2014 , p .93) e.g ., ‘w hen th e op po rtu ni ty to d o it [en ro lli ng in a M O O C] ca m e w ith ou t t he hi nd ra nce of th e lan gu ag e … I d ec id ed to ta ke it ’ ( Re po rt 11 ) N ew E xp eri en ce V iew: In clu de s r efe ren ces to 1) ap pr eci ati on of n ew e xp eri en ces fo r t he n ov elt y f ac tor ---e.g ., ‘ex pl or in g a new co ur se fo rm at I di d no t k no w w as a d et er mi na nt fa ct or fo r t he e nr ol me nt ’ ( Re po rt 44) A cq uai nt an ce s: In clu des ref er en ces to k no w in g so m eo ne 1) p ar tic ip at in g in th e sa me M O O C; 2) wi th pr evi ou s p osi tiv e M O O C ex pe rie nc e; 3) th at e nc ou rag ed e nr olm en t, par tic ipat ion , o r bo th , i n th e M O O C; it ca n c rea te s ome so rt o f l oca l n etwo rk towa rd s M O O C c omp let ion e.g ., ‘lea rn er s w ho e nr ol led in a c ou rs e w ith co llea gu es o r f rie nd s w er e m or e lik el y to be e ng ag ed w ith cou rs e m at er ial s … it shou ld be po ss ib le fo r l ea rn er s t o se lf-id en tify a s a g rou p ta ki ng the c ou rs e to ge th er .’ (K iz ilc ec & S ch ne id er , 2 015, p .6 :17) e.g ., ‘I w as m ad e aw ar e of thi s c ou rs e by c ol le ag ue s … . The ir w or ds of enc ou ra ge m en t … inc ite d m y en ro lm en t.’ (R ep or t 6 6) O rg an is ati on al Fa ct or s: re lat ed w ith the M O O C fo rma t or gan isat io nal fe at ur es th at c an infl ue nc e its su cc es s Sc ale or M as siv en es s: Inc lu de s r efe re nc es to th ou sa nd s o f u ser s b ei ng a bl e to ex pl or e th e M O O C sim ul ta neo us ly du e to 1 ) t he tech no lo gy; 2) the w ay the c ou rs e ope ra te s e.g ., ‘T he ot her m ajo r i ss ue w as th e lack o f co ns id er at io n of th e siz e of the c las se s. For ins ta nc e, so m e cou rs es w ou ld h ave a sin gle di scu ss io n bo ar d fo r t he ten s o f t ho us an ds o f p ar tici pa nt s.’ (W alk-er & L oc h, 2014, p .58) ; O th er a ut ho rs : D ra ke e t a l. (2015 , p .1 33) e.g ., ‘ th e M O O C f or m at m ad e i t p os sib le t o e st ab lis h co m m un ica tio n am on g m an y hu nd red p ar tici pa nt s’ (Re po rt 14 ) O pen nes s: Inc lu de s r efe re nc es to us er s’ fr ee dom to 1 ) e nr ol , a s us ua lly n o p re re qui sit es , f ee s o r com m ut es a re re qu ire d; 2 ) pa rti ci pa te , e .g ., the y ca n sh ar e co nt en t f ro m th e M O O C an d fr om the e xt er ior ; 3 ) l ea ve e.g ., ‘p ar tic ipa nt s of the c ou rs e ar e fr ee to enr ol l or to lea ve a s t he y w ish, … sh ar e co nt ent the y re ce iv ed fr om the c ou rs e w ith ea ch ot her (a nd o ut sid e th e co ur se) , b ut a lso to b rin g in to th e co ur se cont ent the y obt ain ed fr om e lse w he re .’ (D ow ne s, 20 16 , p ar a. 72 ); O th er a ut ho rs : A lra im i e t a l. (2015 , p .33) , R os ew ell & Ja ns en (2014, p .9 3) , S ta ce y (2014 , p .115) e.g ., ‘I en ro lled in th e co ur se , w ith ou t k no w in g fo r sur e i f I w as g oi ng to a tte nd it . I w as im m ed iat ely m ot iv at ed b y [it b ein g] … : v ol un ta ry, o pen , i nt er di s-cipl in ar y, m ul tif ac et ed , d yna m ic , fr ee ’ ( Re por t 1 12 )

SE T OF CA T EG O R IES OF SU CC E SS FA CT OR S AN AL YSI S CA T E G OR IE S O R SU CC E SS FA CT O R S SU PP O R T F R O M T H E L IT E R AT U R E SU PP O R T F R O M T H E E M PI R ICA L D AT A Tech no lo gy : I nc lu de s r efe re nc es to 1 ) t he c ou rs e bei ng su pp or ted b y tech no lo gy ; 2 ) a n ea sy us e of on lin e in fo rm at io n an d co m m un ic at io n tech no l-og ies (e. g. p lat fo rm s) su pp or tin g th e M O O C; 3 ) ha rd wa re a nd so ftwa re su pp ort e.g ., ‘W he n us er s pe rc eiv e th at the M O O C pl at for m pr ov id es a fu lly fu nct io ni ng sys tem fo r l ea rn in g, th eir co nt in ua nce in te nt io ns to w ar d pa rti cip at io n in M O O Cs w ill b e po sit iv e in flu en ced ’ ( Y an g et a l., 2 01 7, p .6 ); O th er a ut ho rs : G am ag e et a l. (201 5, p .227) e.g ., ‘H ow ev er , t he pl at fo rm w as so m ew ha t co nf us in g an d co un ter -in tu iti ve .’ (Re po rt 14 4) Fu nd in g: Inc lu de s r efe re nc es to 1) M O O C pr ov id er s’ need to fi nd fu nd in g op tio ns o th er th an u ser s’ tu iti on fee s e.g ., ‘th ey [in itia tive s o f o pen ed uca tio n] do n ot ch ar ge us er s f or ta ki ng co ur ses o r u sin g m at er ial s [s o th ey need to ] f in d fea sib le fu nd in g st ra te gie s t o m ain ta in su st ain ab ilit y.’ (N ag as him a, 201 4, p. 16 ); O th er a ut ho rs : P oy & G on za les -A gu ila r ( 2014, p .110) ---Cr ed its R ec og ni tio n: Inc lu de s r efe re nc es to 1) cr ed its a ttr ib ut io n fo r M O O C acco m pl ish -me nt e.g ., ‘L ea rne rs w ho re por te d the int ent ion to ea rn a ce rti fic at e w er e no t m or e lik ely to a ct ua lly ea rn a c er tif ica te th an th os e w ho d id n ot in ten d to ea rn o ne – de spi te the fa ct tha t t he y w er e m or e lik ely to at te m pt a ss ig nm en ts .’ (K iz ilc ec & S ch ne id er , 20 15, p .6: 20) ; O th er au th or s: Ro se w ell & Ja nse n (2 01 4, p .9 3) , Y ep es -Ba ld ó et a l. (2016 , p. 190) e.g ., ‘T he e xp ec ta tions c re at ed w ith thi s M O O C ar e lin ked n ot o nl y to … , b ut a lso to th e po ss ib ili ty of do in g an o nl in e cer tif ied co ur se ’ ( Rep or t 5 3) Pe da gog ic al Fa ct or s: re lat ed w ith M O O Cs p ro -vid er s’ te ac hin g de cisi on s t ha t ca n inf lu enc e its su cc es s Foc us of Su bj ec ts : I nc lu de s off er in g 1) c ou rs e th em es o f p er so na l o r p ro fes sio na l i nt er es t f or spe cif ic ta rg et g rou ps ; 2 ) i nn ov ati ve t he me s ( e.g . in ter dis cip lin ar y). In th e c as e o f t he M O O C ‘ As alt er aç õe s c lim át ic as n os m éd ia es co lar es - Cl i-m a@ Ed uM ed ia’ the su bj ec ts a re c lim at e cha ng e, m ed ia an d te ac hi ng e.g ., ‘D ec id ing w hi ch su bj ec t t o off er is a fa ct or ine vi ta bl y affe ct ing th e su cces s o f o pe n ed uca tio n m ov em en ts , w hi ch is d riv en b y lea rn er s’ re as on s f or ta ki ng co ur ses … a cq ui rin g kn ow led ge fo r th eir jo bs [i n sci en ce -re lat ed co ur se s o r] ... p er so na l i nt er es t [ in hu m an itie s]’ (N ag as him a, 2014, p .1 7) e.g ., ‘m y exp ec ta tio ns w er e, also , t o lea rn a bo ut m ed ia pr od uc tio n a nd m ed ia us e i n t he c las sr oo m , a s w ell a s to d ev elo p sk ill s t o tea ch cl im at e ch an ge in a n in no-va tiv e and e ffe ct iv e w ay .’ (R epor t 3 3) Pr e-Co ur se In fo rm at io n: In clu des p ro vi di ng in fo rm at io n bef or e en ro lm en t, reg ar di ng 1 ) t he M O O C’ s o bj ect iv es a nd co nt en t; 2) th e w ay it op er at es e.g ., ‘cl ea r d ec lar at ion of w ha t s or t of cou rs e the y [pot ent ial re gi s-tra nt s] a re si gn in g up fo r’ ( Cr ee lm an e t a l., 2 01 4, p .8 6) ; O th er au th or s: A lra im i e t a l. (20 15, p .35) , W alke r & L oc h (201 4, p .59) ; Ye pe s-Ba ld ó et a l. (2016 , p .19 0) . e.g ., ‘In ter m s o f e xp ect at io ns , t he pr es en ta tio n vi deo all ow ed us to g ue ss th e g re at qua lit y of th e w or k’ (R ep or t 117) Tim in g: Inc lu de s m ee ting a ud ienc es ne ed s i n te rm s o f 1 ) s ch ed ul e; 2 ) d ur at io n; 3 ) s elf -m ana gi ng the ti m e ne ed ed for the c ou rs e e.g ., ‘C oh er enc e and a de qu ac y of the ti m ing ’ ( Ye pe s-Bal dó e t al ., 2016, p .190) ; O th er a ut ho rs : G am ag e et a l. (2015, p .227) e.g ., ‘T he po ss ib ili ty of … a ut on om ou sly m an ag in g th e nu m ber a nd m om en t o f d ed ica ted h ou rs w as a lso an ot he r f ac to r t ha t mo tiv at ed me to p ar tic ip at e in th is co ur se. ’ ( Rep or t 2 9) Co nt en t O rg an isa tio n an d A cces s: Inc lu de s re fe re nc es to the w ay the se le ct ion and st ru ct ur e of the c ou rs e co nt ent , e .g ., 1) m aki ng co nt en t av ail ab le i n a t im ely m an ne r; 2) o rg an isi ng co nt en t in a w ay th at a llo w us er s t o e as ily fi nd th e d e-sir ed in fo rm at io n an d le ar n; 3 ) pr ov idi ng cle ar ins tru cti on s o n h ow t o u se t he M O O C t o a ch iev e l ea rn ing go als ; 4 ) b ala nc in g th e req ui red w or kl oa d to a vo id ove rlo ad . e.g ., ‘L ea rn er s f or w ho m co ur se co nt en t w as rel ev an t t o th eir cu rr ent a ca de m ic end ea vor s … u se the e nv ironm ent a s a re fe re nc e so urc e. … Bre ak in g ap ar t M O O C co nt en t i nt o sep ar at ely ta gg ed m od ul es w ou ld a llow re fe re nc e-st yl e us ag e.’ (K iz ilc ec & S chne id er , 2015, p .6: 19) ; O th er a ut ho rs : G am ag e et a l. (2015 , p .22 7) ; K en ne dy (2014, p .8) ; Y ep es -B ald ó et a l. (20 16, p .190) e.g ., ‘I also h ig hl ig ht th e qu ali ty o f t hi s c ou rse ’s st ru c-tu re, fr om a g lo ba l a pp ro ach to a m or e sp eci fic on e.’ (R ep or t 102)

SE T OF CA T EG O R IES OF SU CC E SS FA CT OR S AN AL YSI S CA T E G OR IE S O R SU CC E SS FA CT O R S SU PP O R T F R O M T H E L IT E R AT U R E SU PP O R T F R O M T H E E M PI R ICA L D AT A Q ua lit y Re sou rc es : I ncl ud es p ro vi di ng lea rn in g ma te ria ls 1) w ith re lev an t a nd u pd at ed c on te nt ; 2) d es ig ne d spe cifi ca lly for the onl ine for m at , w ith c og nit ive a nd me ta -c og nit iv e st im uli, sh or t lect ur es o n sin gl e to pi cs a nd lea rn in g or ga ni ser s e.g ., ‘C ou rs es sh ou ld p ro vi de hi gh q ua lit y m at er ial s t o en ab le an in dep en den t l ea rn er to p ro gr es s t hr ou gh sel fs tu dy. ’ ( Ro sew ell & Ja ns en , 2 01 4, p .9 3) ; O th er a ut ho rs : D ra ke et a l. (2015 , p .1 31 -13 2) , G am ag e et a l. (2015 , p .22 7) ; W alke r & L oc h (2014, p .5 8) , Y an g et al. (2017 , p .6) , Y ep es -B ald ó et a l. (2016, p .190) e.g ., ‘th e pr es en ted co nt en t a nd res ou rces h av e a hi gh qu alit y.’ (R ep or t 123) Fle xib ilit y an d Sc af fo ld in g fo r D ive rs ity : I n-clu des p ro vi di ng 1 ) a ra ng e of p os sib le le ar ni ng pa th s i n th e M O O C (e. g. , ex pl or in g co nt en t i n a di ffe re nt or de r or onl y pa rti all y) , a cc or di ng ly to the u se rs ’ n ee ds ; 2 ) s ca ffol di ng sk ill s ne ed ed for M O O C exp lo ita tio n, su ch a s se lf-or gan isat io n an d tech no lo gy sk ill s e.g ., ‘th e im pr ov em ent of the q ua lit y of M O O C offe ring d epe nd s on h ow m uch m or e fle xi bl e an d ad ju st ab le to d iff er en t co nt ex ts an d need s t he lea rn in g op po rtu ni ties p ro vi ded ca n be co m e. … th e re al su cc es s f ac tor in a M O O C is the le ve l of eng ag em ent obt aine d fr om c ou rse p ar tic ip an ts .’ (H er ná nd ez e t a l., 2 01 4, p .1 40 ); O th er au th or s: Cr ee lm an e t a l. (2014, p .8 5) , D eb oe r e t a l. (2014, p .82) , D ra ke e t a l. (201 5, p .133) , K en ne dy (2014, p .8) , K iz ilc ec & S ch ne i-de r ( 2015, p .6: 17) , Po y & G on za le s-A gu ila r ( 2014, p .110) , Ro se w ell & Ja ns en (2014 , p .9 3) , Y an g et a l. (2017, p .6) , Y ep es -Ba ld ó et a l. (2016 , p .1 90) e.g ., ‘ I p ar tic ip at ed in th e M O O C f or … , t he le ar ni ng au to no m y an d … ’ ( Rep or t 1 9) In te ra ct ivit y and Pe er -to -Pe er Pe da gog y: In -clu des p ro vi di ng feed ba ck o n lea rn in g (ei th er fr om th e pr ov id er o r p eer s) th ro ug h 1) int er ac tiv e and ri ch m ed ia re sou rc es , s uc h as int er ac tiv e vi de os or se lf-gr ad in g qu iz ze s; 2) co m m un ica tio n st ra teg ies a bo ut th e lea rn in g co nt en t ( co lla bo ra tio n) , s uch a s d iscu ss io n pr om pt s f or fo ra , l oca l s tu den t m eet in gs o r p eer asse ssm en ts e.g ., ‘C ou rse m at er ial s sh ou ld m ak e be st u se o f o nl in e af fo rd an ce s (in te ra ct iv ity , c om m un ic at io n, c oll ab or at io n) a s w ell as ric h m ed ia (v id eo a nd a ud io ) t o en ga ge st ud en ts w ith th eir lea rn in g. ’ ( Ro sew ell & Ja nse n, 2 01 4, p .9 3) ; O th er a ut ho rs: C re elm an e t a l. (2 01 4, p .8 6) , D ow ne s ( 2016 , p ar a. 75) , D ra ke e t a l. (2015 , p .13 2) , G am ag e et a l. (2015, p .2 27) ; H er ná nd ez e t a l. (20 14, p .14 0) , N ag as him a (2014 , p. 13, 17) , R oo ij & Z irkle (20 16, p .5 ), St ac ey (2014 , p .11 5) , W alke r & L oc h (2014, p .5 9) , Ye pe s-Ba ld ó et a l. (2016 , p .190) e.g ., ‘I em ph as ise the a dd ed v alu e of the for a, w hi ch all ow ed th e di scu ss io n of id ea s, th e ex pl an at io n of so m e issu es, a nd in fo rm at io n ex ch an ge’ (R ep or t 3 9)

Azevedo & Marques

243 Moreover, as the aim was to identify the presence and direction of influence of each success factor, only one occurrence was registered when a factor was recognised in more than one unit, within the same case. For example, Report 4 contained two different units recordable into the category ‘Quality Resources’ and was coded with ‘+1’: “I highlight the extension manuals as a resource that allowed me to better understand the concept of …” and “I analysed the proposals of teaching strategies, pro-posed by the MOOC team, with interest and verified they are well structured … .”

Two coders were trained in the tasks of unitizing, coding and making records in the SPSS. The train-ing included familiarization with the analysis framework and collective open codtrain-ing of a convenience sample of 15 cases. The coding process was discussed and both, the coding instructions and analysis framework, were refined. An independent coding of a convenience sample of 20 cases was per-formed, to refine further the analysis framework and ascertain the quality of the coding process. To determine the inter-rater reliability of nominal variables the Krippendorff ’s alpha (α) or kalpha (Hayes & Krippendorff, 2007) is a suitable reliability index (Taylor & Watkinson, 2007). Hence, the kalpha coefficients for the success factors, regarding the initial coding, are presented in Graphic 1.

Graphic 1. MOOC success factors’ kalpha coefficients for the 20 cases of the initial independent coding sample.

Krippendorff (2004b) recommends considering α < 0.8 as good reliability and 0.667 ≤ α ≤ 0.8 for drawing tentative conclusions. Six success factors presented no variation between coders. ‘Credits Recognition’ (α = 0.6486) and ‘Quality Resources’ (α = 0.6549) were slightly below the cut limit, so, considering the high subjectivity associated with a semantic approach to content analysis, independ-ent coding of the remaining 124 cases was performed with the refined analysis framework.

This study complies with research ethics norms from the Research Ethics Guidebook, such as the voluntary participation, confidentiality, and participants’ anonymity.

New Experience View

MOOC Success Factors

244

S

’

The authors acknowledge some limitations in this study, including:

• the possibility of bias in the teachers’ testimonies, as they were produced in the context of an accreditation process;

• _{the limited amount of reports, or sample (159, when the MOOC was finished by 311 users),} hinders generalization of results, although offering relevant issues to consider even in differ-ent contexts; and

• _{the fact that this content analysis used data from only one MOOC.}

Further research should include more data, e.g., data collected in other MOOC initiatives to check the framework adequacy against bigger samples in educational contexts, and should collect data pro-duced outside an accreditation context as well to reduce the possible bias caused by this option. Ad-ditionally, this analysis framework should be the object of further empirical-led refinement to include reliable measurements of the participants’ learning, associated with their MOOC involvement, as in this study this was not possible: the collected data gave the authors access to the participants’ per-spectives about their learning in this MOOC, but not to their actual learning. The refinement of this analysis framework should also include data from other (non-educational) contexts, to allow attempts of generalisation across contexts of study. Nevertheless, this study is a first step into the develop-ment of a useful analytical framework that will be able to contribute to the literature and will allow proposing a set of empirically based recommendations for MOOC providers.

P

RESENTATION AND

D

ISCUSSION OF THE

R

ESULTS

A hundred and twenty-four teacher reports were considered in the main analysis. Inter-rater reliability was computed and presented in Graphic 2.

Graphic 2. MOOC success factors’ kalpha coefficients for the 124 cases used in the independent coding.

Similarly to the initial analysis, in the main one no units were coded in ‘Funding’. Therefore, this factor seems to be absent from the participants’ minds. Nevertheless, it can be a crucial factor for

New Experience View

Azevedo & Marques

245 MOOC providers (Nagashima, 2014). This result can be interpreted as supporting the claim that different MOOC stakeholders have different success and quality criteria (e.g., Downes, 2016; Walker & Loch, 2014).

‘Learning view’ and ‘Flexibility and scaffolding for diversity’ presented an α value well below 0.667, hence, these literature-based success factors were not considered for further analysis due to insuffi-cient reliability.

With kalpha coefficients only slightly below 0.667, ‘Localisation’ (α = 0.6639), ‘Pre-course infor-mation’ (α = 0.6577), ‘Content organisation and access’ (α = 0.5740), and ‘Quality resources’ (α = 0.5715) were considered for further analysis, with reservations.

In the 124 reports considered for analysis, a total of 594 references (see Table 2) were made to fac-tors influencing positively or negatively the enrolment, the MOOC continuance, users’ learning or all of these. It was possible to identify from one to ten factors in each report; however, 59.0% men-tioned four or five success factors, and 93% menmen-tioned between three and six factors. The distribu-tion of the number of factors found in each report is presented in Graphic 3.

Graphic 3. Number of influential factors per case

Table 2 presents a synthesis of the frequency and of the relative importance of the considered MOOCs’ success factors. Noticeable, when a factor was mentioned, it usually had a positive effect (e.g., ‘Technology’ was reported as a positive influence in 31 reports, negative in 12, and contradicto-ry in eight). The references were distributed in the three sets of categories of analysis (social, organi-sational, and pedagogical). In this table, for each set, we used italics for the success factor with the highest percentage of cases occurrence.

Table 2. Frequency and percentage of MOOCs’ success factors, present in the collected data.

SOCIAL

FACTORS ORGANISATIONAL FACTORS PEDAGOGICAL FACTORS _TOTAL R or

B Lo* NE Ac S or M Op T CR FS* PI* Ti CO* QR* IP

Influence + _- 16 _{0 0} 1 23 10 _{1 0 10} 4 14 _{0 12} 31 17 117 _{0 0} 3 ₂ 60 ₁ 53 115 74 538 _{7 0 0 33} 594

contrad. 0 0 0 0 0 0 8 0 0 0 3 8 1 3 23

No influence 108 123 100 114 110 110 73 107 7 119 60 56 8 47 1142

% of cases with

influ-ence 12.9 0.8 19.4 8.1 11.3 11.3 41.1 13.7 94.4 4.0 51.6 54.8 93.5 62.1 % of Cases 32.3 54.0 100.0

MOOC Success Factors

246

‘Social factors’ were referred by almost one-third of the teachers (32.3%); hence, this set of factors is fairly present in their minds. In this set of categories, the data-based factor ‘New experience view’ was the most influential one, with 19.4% of references. It was followed by the literature-based ‘Repu-tation or brand’, with 12.9%. These results show the relevance that both literature and empirical data have in an effort to identify success factors and, hence, good practices. This cohort of teachers seems to value new experiences, particularly when they can show them first-hand new ways of teach-ing and learnteach-ing. Additionally, as reported in the literature (Alraimi et al., 2015; Nagashima, 2014), the way MOOC providers are perceived by the media and the public does affect the enrolment rates. This claim is reinforced by the high enrolment numbers in MOOCs provided by high profile Ameri-can universities (Ho et al., 2015; Ho et al., 2014). Contrasting, ‘Localisation’ arises as a clearly no influential factor for this cohort of users.

The ‘Organisational factors’ seem more relevant for this cohort of users, as 54.0% mentioned the influence of these factors. Remarkably, ‘Technology’ was the most relevant one, with mentions in 41.1% cases. However, opinions were divergent, as positive references were frequently made to the platform’s intuitiveness of use and negative ones were made to technical problems encountered dur-ing navigation. This technical factor is an important one to consider by MOOC providers, as partici-pants expressed expectations of spending their online time reaching their own goals, and not dealing with uneasy distance communication tools. The following citation illustrates: “I didn’t find the plat-form intuitive, as I spent a lot of time learning how to use it. I wrote comments in the forum that were lost twice” (Report 14).

On the other hand, the literature’s controversy regarding credit attribution in formal higher education (Creelman et al., 2014; Downes, 2016; St Clair et al., 2015) did not have an echo in our empirical data, as only 13.7% of the participants mentioned this factor. Even though this course was developed in a continuous teacher professional development context, rather than in a higher education one, this is a surprising result, as all the reports were written as a requirement to obtain a certification relevant in the Portuguese education system.

Finally, all the teachers pointed the influence of at least one ‘Pedagogical factor’ (100%). This result is not surprising for this cohort of teachers as MOOC participants. With 94.4% of mentions, ‘Focus of subjects’ is the most influential factor of all, closely followed by ‘Quality resources’, with 93.5%. The high frequency of references to personal or professional interest in the MOOC’s topic was expected. In our users’ testimonies, this influence was always positive: “The enrolment and attendance of this MOOC resulted from the interest and topicality of the theme, as it refers to a curricular topic of the subject I’m teaching … that concerns us all” (Report 26). ‘Quality resources’ was another clearly influential literature-based factor (e.g., Drake et al., 2015). Participants were satisfied with the materi-als they accessed in this MOOC and, being teachers, frequently mentioned their utility in their own classrooms: “The videos were enlightening and engaging. They are great resources for classroom use” (Report 126). Another example of teachers as MOOC participants using resources in their classrooms was reported in the literature and the potential of this course format for “disseminating course tools, pedagogical innovations, and teaching modules” (Ho et al., 2015, p.5) was acknowl-edged.

Also, worth noting with high percentage of mentions are the success factors ‘Interactivity and Peer-to-Peer Pedagogy’ (in 62.1% reports), ‘Content Organisation and Access’ (in 54.8%), and ‘Timing’ (in 51.6%).

To understand the relative importance of each MOOC’s success factor category, in the analysed data, the percentage of cases in each one, distributed by number of mentions in each report, was comput-ed and presentcomput-ed in Graphics 4, 5 and 6.

Azevedo & Marques

247

Graphics 4, 5 and 6. Relative importance of each MOOCs’ success factor category (Social, Organisational and Pedagogical, respectively), in the empirical data analysed

Graphic 6 highlights the higher importance of ‘Pedagogical factors’, when comparing with ‘Organi-sational’ (Graphic 5) and ‘Social’ (Graphic 4) ones, for this cohort of teachers. Most did not acknowledge the influence of ‘Social factors’ (67.7%) or mentioned only one (25.0%). Similarly, but with higher percentages, the ‘Organisational factors’ were usually referred to only one time (34.7%) or none at all (46.0%). Contrasting with these scenarios, the ‘Pedagogical’ factors were usually men-tioned four (38.7%) to three (29.0%) times per report. This result illustrates the importance of not neglecting pedagogical factors, particularly when the MOOC is targeting teachers as users.

Finally, the presented and discussed data did not give access to an important MOOC success factor: the participants’ learning; however, they provide an indicator of this factor: the participants’ perspec-tive regarding their learning within the MOOC. Hence, the analysed data allowed to propose a framework for MOOC’s success analysis, as they provided insights into the participants’ enrolment motivations, continuance factors and perceived learning within the MOOC.

C

ONCLUDING

R

EMARKS

The literature does not present many studies on MOOC success factors. Presenting simple generic statements regarding MOOCs can be counterproductive due to their complexity and diversity (St Clair et al., 2015). For example, as different stakeholders are differently influenced by different issues, the aim of identifying relevant success factors becomes a challenging one. Nevertheless, to make some sense of the MOOC phenomenon and to support MOOC providers when designing, review-ing or runnreview-ing their courses, both the literature and empirical data are sources of good practices. This claim is supported by other authors (e.g., Yepes-Baldó et al., 2016). In this study, the authors present a MOOC success factors framework that was literature-based and empirically refined through

MOOC Success Factors

248

content analysis, in a qualitative interpretivist research approach. Among the identified set of MOOC success factors, the authors highlight the following (in order of relevance in the analysed data):

• ‘Focus of subjects’ seems to positively influence the participants’ enrolment in a MOOC, if they are interested in its topics, for personal or professional reasons, which can be reinforced when those topics are perceived as innovative;

• _{‘Quality Resources’ seems to positively influence the participants’ continuance and learning} in a MOOC, as they are perceived as relevant and updated, as well as adequate for the online format;

• _{‘Interactivity and Peer-to-Peer Pedagogy’ also seems to positively influence the participants’} continuance and learning in a MOOC, particularly when feedback on learning is provided, either by interactive and rich media resources or by collaborative communication about the learning content;

• ‘Content Organisation and Access’ seems to positively influence the participants’ continu-ance and learning in a MOOC as well, as they perceive, for example, that they have timely access to desired content and are able to easily find desired information; and

• ‘Timing’ seems to influence both enrolment and continuance in a MOOC, as the partici-pants’ needs, in terms of schedule, duration and self-managing of the time required for the course, are met.

Although the low barriers to enrolment and the asynchronous running of MOOCs typically encour-age diversity in enrolees’ intentions and actions (DeBoer et al., 2014), the analysed MOOC targeted a specific audience: Portuguese school teachers, interested in climate change and media. This cohort certainly influenced our results.

Our literature review and data analysis allowed us to identify a set of factors influencing the enrol-ment, continuance and learning of MOOC participants. Following Nagashima (2014), these success factors were organised into three categories: ‘Social’, ‘Organisational’ and ‘Pedagogical’. This last type of factor was clearly the most influential for this cohort of teachers, particularly the MOOC’s subject and resources. However, other features also seem relevant, such as the interactivity among partici-pants, the content organisation and access, and the course’s timing. This factor seems to influence MOOCs’ success, particularly for target groups with a professional life with peaks and lows of work-load along the year, such as teachers. This issue was reported before (Marques, Loureiro, & Marques, 2016).

Noting that most of the highly valued ‘Pedagogical factors’ were based on the literature, some data-emergent factors were of social and organisational nature; hence, their relevance cannot be dimin-ished. More specifically, while MOOCs are considered new positive experiences, the novelty factor can be a good stimulus for enrolment.

A

This paper is published as part of the project Clima@EduMedia, which was co-financed by the EEA Grants at a rate of 85% and 15% by the Portuguese Environment Agency, IP (APA, IP) through the Portuguese Carbon Fund. The authors would like to thank Diana Seabra for her contributions to the data analysis.

Azevedo & Marques

249

R

EFERENCES

Admiraal, W., Huisman, B., & Pilli, O. (2015). Assessment in massive open online courses. Electronic Journal of e-Learning, 13(4), 207-216.

Alraimi, K.M., Zo, H., & Ciganek, A.P. (2015). Understanding the MOOCs continuance: The role of openness and reputation. Computers & Education, 80, 28-38. doi:10.1016/j.compedu.2014.08.006

Anderson , T. (2016). Theories for learning with emerging technologies. In G. Veletsianos (Ed.), Emergence and innovation in digital learning: foundations and applications (pp. 35-50). Edmonton: Athabasca University Press Bates, T. (2014, November 16). Why the fuss about MOOCs? Political, social and economic drivers. [personal website

post]. Retrieved from http://www.tonybates.ca/2014/11/16/why-the-fuss-about-moocs-political-social-and-economic-drivers/

Christensen, G., Steinmetz, A., Alcorn, B., Bennett, A., Woods, D., & Emanuel, E.J. (2013, April 18). The MOOC phenomenon: Who takes massive open online courses and why? [working paper]. Retrieved from http://ssrn.com/abstract=2350964

Creelman, A., Ehlers, U.-D., & Ossiannilsson, E. (2014). Perspectives on MOOC quality – An account of the EFQUEL MOOC Quality Project. International Journal for Innovation and Quality and in Learning

(INNOQUAL), September(3), 78-87.

DeBoer, J., Ho, A. D., Stump, G. S., & Breslow, L. (2014). Changing “course”: Reconceptualizing educational variables for massive open online courses. Educational Researcher, 43(2), 74-84.

doi:10.3102/0013189X14523038

Downes, S. (2016). The quality of massive open online courses [personal page post]. Retrieved from http://www.downes.ca/post/66145

Drake, J. R., O'Hara, M., & Seeman, E. (2015). Five principles for MOOC design: With a case study. Journal of Information Technology Education: Innovations in Practice, 14, 125-143. Retrieved from

https://www.informingscience.org/Publications/2250

Gamage, D., Fernando, S., & Perera, I. (2015). Quality of MOOCs: A review of literature on effectiveness and quality aspects. 2015 8th _{International Conference on Ubi-Media Computing, UMEDIA 2015 - Conference}

Proceed-ings, 224-229. doi:10.1109/UMEDIA.2015.7297459

Hayes, A.F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods & Measures, 1(1), 77. doi:10.1080/19312450709336664

Hernández, R., Morales, M., Mota, J., & Teixeira, A. (2014). Promoting engagement in MOOCs through social collabora-tion: Common lessons from the pedagogical models of Universidad Galileo and Universidade Aberta. Paper presented at the 8th _{EDEN Research Workshop. Challenges for Research into Open & Distance Learning: Doing}

Things Better: Doing Better Things, Oxford, United Kingdom.

Ho, A. D., Chuang, I., Reich, J., Coleman, C. A., Whitehill, J., Northcutt, C. G., ... Petersen, R. (2015). HarvardX and MITx: Two years of open online courses, Fall 2012-Summer 2014. Retrieved from

http://dx.doi.org/10.2139/ssrn.2586847

Ho, A.D., Reich, J., Nesterko, S., Seaton, D. T., Mullaney, T., Waldo, J., & Chuang, I. (2014). HarvardX and MITx: The first year of open online courses, Fall 2012-Summer 2013. Retrieved from http://ssrn.com/abstract=2381263 Jordan, K. (2015). Massive open online course completion rates revisited: Assessment, length and attrition.

International Review of Research in Open & Distance Learning, 16(3), 341-358. doi:10.19173/irrodl.v16i3.2112 Kady, H. R., & Vadeboncoeur, J. A. (2013). Massive open online courses (MOOC). Salem Press Encyclopedia. Kennedy, J. (2014). Characteristics of massive open online courses (MOOCs): A research review, 2009-2012.

Journal of Interactive Online Learning, 13(1), 1-15.

Kizilcec, R.F., & Schneider, E. (2015). Motivation as a lens to understand online learners: Toward data-driven design with the OLEI scale. ACM Transactions on Computer-Human Interaction, 22(2), 6:1-6:24.

MOOC Success Factors

250

Kovanović, V., Joksimović, S., Gašević, D., Siemens, G., & Hatala, M. (2015). What public media reveals about MOOCs: A systematic analysis of news reports. British Journal of Educational Technology, 46(3), 510-527. doi: 10.1111/bjet.12277

Krippendorff, K. (2004a). Content analysis: an introduction to its methodology (2nd ed). Thousand Oaks, CA: Sage Publications.

Krippendorff, K. (2004b). Reliability in content analysis: Some common misconceptions and recommenda-tions. Human Communication Research, 30(3), 411-433. doi:10.1111/j.1468-2958.2004.tb00738.x

Liyanagunawardena, T. R., Adams, A. A., & Williams, S. A. (2013). MOOCs: A systematic study of the pub-lished literature 2008-2012. International Review of Research in Open and Distance Learning, 14(3), 202-227. Marques, M. M., Loureiro, M. J., & Marques, L. (2016). The dynamics of an online community of practice

involving teachers and researchers. Professional Development in Education, 42, 235-257. doi: 10.1080/19415257.2014.997396

Nagashima, T. (2014). What makes open education thrive? Examination of factors contributing the success of open education initiatives. International Journal for Innovation and Quality and in Learning (INNOQUAL), Sep-tember(3), 10-21.

Poy, R., & Gonzales-Aguilar, A. (2014). MOOC success factors: Some critical considerations. RISTI - Revista Iberica de Sistemas e Tecnologias de Informacao, 1(E1), 105-118. doi:10.4304/risti.e1.105-118

Rooij, S., & Zirkle, K. (2016). Balancing pedagogy, student readiness and accessibility: A case study in collabo-rative online course development. Internet and Higher Education, 28, 1-7. doi:10.1016/j.iheduc.2015.08.001 Rosewell, J., & Jansen, D. (2014). The OpenupEd quality label: Benchmarks for MOOCs. International Journal for

Innovation and Quality and in Learning (INNOQUAL), September(3), 88-100.

Saadatdoost, R., Sim, A., Jafarkarimi, H., & Mei Hee, J. (2015). Exploring MOOC from education and infor-mation systems perspectives: A short literature review. Educational Review, 67(4), 505-518.

doi: 10.1080/00131911.2015.1058748

Siemens, G. (2005). Connectivism: A learning theory for the digital age. International Journal of Instructional Tech-nology and Distance Learning, 2(10). Retrieved from http://www.itdl.org/journal/jan_05/article01.htm St Clair, R., Winer, L., Finkelstein, A., Fuentes-Steeves, A., & Wald, S. (2015). Big hat and no cattle? The

impli-cations of MOOCs for the adult learning landscape. Canadian Journal for the Study of Adult Education, 27(3), 65-82.

Stacey, P. (2014). Pedagogy of MOOCs. International Journal for Innovation and Quality and in Learning (INNOQUAL), September(3), 111-115.

Taylor, J., & Watkinson, D. (2007). Indexing reliability for condition survey data. The Conservator, 30(1), 49-62. doi:10.1080/01410096.2007.9995223

Twining, P., Heller, R. S., Nussbaum, M., & Tsai, C.-C. (2017). Some guidance on conducting and reporting qualitative studies. Computers & Education, 106, A1-A9. doi:10.1016/j.compedu.2016.12.002

Walker, L., & Loch, B. (2014). Academics’ perceptions on the quality of MOOCs: An empirical study. Interna-tional Journal for Innovation and Quality and in Learning (INNOQUAL), September(3), 53-63.

Yang, M., Shao, Z., Liu, Q., & Liu, C. (2017). Understanding the quality factors that influence the continuance intention of students toward participation in MOOCs. Educational Technology Research and Development, 1-20. doi:10.1007/s11423-017-9513-6

Yepes-Baldó, M., Romeo, M., Martín, C., García, M. A., Monzó, G., & Besolí, A. (2016). Quality indicators: Developing “MOOCs” in the European Higher Education Area. Educational Media International, 53(3), 184-197. doi:10.1080/09523987.2016.1236998

Azevedo & Marques

251

B

IOGRAPHIES

José Azevedo is Associate Professor at the Faculty of Arts of the

Uni-versity of Porto, Portugal

(https://sigarra.up.pt/flup/pt/func_geral.formview?p_codigo=215791). He is maintaining a dual career, focusing simultaneously on research and on audio-visual contents production, with the aim of raising public awareness of science and science literacy. Some of the most recent pro-jects he coordinated are Science 2.0 (www.ciencia20.up.pt) (2011-2013), Clima@EduMedia – Climate Change: learning through the school media (www.climaedumedia.com) (2014-2016), and Nutriscience: play, cook and learn (https://nutriciencia.pt/) (2014-2016).

Margarida Morais Marques holds a PhD in Didactics and Training and

is currently a research fellow at the EduPARK project, which is creating of a mobile interactive Augmented Reality (AR) application for interdis-ciplinary learning in Science Education. She has worked in Climate Change Education in the Clima@EduMedia project, namely in the pro-ject’s MOOC, and her main research interest is in the use of technologies and gamification strategies in the promotion of Scientific Literacy.