Evaluation of the Application of e-Learning Methodologies to the Education of Engineering

Texto

(1)Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. EVALUATION OF THE APPLICATION OF ELEARNING METHODOLOGIES TO THE EDUCATION OF ENGINEERING. RITA RODRIGUES CLEMENTE FALCÃO DE BERREDO Dissertação submetida para satisfação dos requisitos do grau de DOUTOR EM MEDIA DIGITAIS ESPECIALIDADE DE CRIAÇÃO DE AUDIOVISUAL E DE CONTEÚDOS INTERACTIVOS. Orientador: Professor Doutor Alfredo Augusto Vieira Soeiro. AGOSTO DE 2012. I.

(2) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. II.

(3) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. Versão revista em Abril 2013 Dissertação desenvolvida no âmbito do Programa Doutoral em Media Digitais da Faculdade de Engenharia da Universidade do Porto em colaboração com a Universidade do Texas em Austin. Revised version, April 2013 Dissertation submitted to the Doctoral Programme in Digital Media at the School of Engineering of University of Porto, in collaboration with University of Texas in Austin. III.

(4) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. IV.

(5) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. ACKNOWLEDGMENTS I want to express my sincerest gratitude to Professor Alfredo Soeiro for encouraging and supervising my research. His constant support, guidance and the several productive discussions were essential for reaching this final stage. My sincerest appreciation is extended to Professor Gráinne Conole and Professor Joan Hughes for the useful conversations and their activity in my committee. I am also grateful to Professor Raul Vidal, Professor Rui Maranhão and Professor José Fernando Oliveira for their willingness to participate in this research study. I am especially thankful to Professor Gabriel David for his willingness to participate in the study and his continuous disposition to help in the development of the technical component of the project. I would also like to acknowledge my appreciation to Dra. Lígia Ribeiro and to all the members of the Unit for New Technologies in Education of Universidade do Porto for the constant support in my pursuit to extend my studies. My gratitude also goes to the Fundação para a Ciência e Tecnologia for their financial aid and the opportunity to research the area of e-learning and assessment. My appreciation also goes to my closest friends and family for their support and understanding that helped me to overcome the difficulties created by research work. A very special thanks goes to Nuno, for being there, for his support, encouragement and patience, and especially for his commitment to my researcher’s life. Finally, my sincerest gratitude goes to my Mother that was always there, supporting me in every possible way during this long period and in particular for being the best Grandmother that Francisco could ever wished to have. To Francisco, my son, I dedicate this work.. Your education is the only thing that you will always carry with you, no matter what happens. Nobody can take it away. Luis Falcão de Berredo. To you, my Father, only one word: saudade…. V.

(6) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. VI.

(7) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. TABLE OF CONTENTS ACKNOWLEDGMENTS ..................................................................................................................... V TABLE OF CONTENTS ................................................................................................................... VII ABSTRACT ......................................................................................................................................... XI RESUMO .......................................................................................................................................... XIII RESUME ............................................................................................................................................. XV LIST OF TERMS AND ACRONYMS IN USE ............................................................................. XVII LIST OF TABLES .............................................................................................................................XIX LIST OF FIGURES ...........................................................................................................................XXI CHAPTER 1 - INTRODUCTION.................................................................................................... 1 1.1. Background ......................................................................................................................................... 1. 1.2. Field of Study and Intended Stakeholders ........................................................................................... 2. 1.3. Background and motivation of the researcher .................................................................................... 3. 1.4. Statement of the problem and research questions.............................................................................. 4. 1.5. Impact expectations of the study ........................................................................................................ 5. 1.6. Structure of the study ......................................................................................................................... 6. 1.7. Publications related with the research study ...................................................................................... 6. CHAPTER 2 - RELATED WORK................................................................................................... 9 2.1. Introduction ........................................................................................................................................ 9. 2.2. Assessment ........................................................................................................................................11. 2.3. Overview on e-assessment .................................................................................................................30. 2.4. Learning Outcomes ............................................................................................................................32. 2.5. The alignment question......................................................................................................................39. VII.

(8) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. 2.6. Learning Outcomes in engineering education .................................................................................... 41. 2.7. Conclusions from the literature review .............................................................................................. 45. CHAPTER 3 - RESEARCH METHOD ........................................................................................ 47 3.1. Research design ................................................................................................................................. 47. 3.2. Development of the conceptual model .............................................................................................. 48. 3.3. Multiple case-studies approach ......................................................................................................... 52. 3.4. Data collection ................................................................................................................................... 52. 3.5. Data analysis ...................................................................................................................................... 54. 3.6. Validity, reliability and limitations ..................................................................................................... 55. 3.7. Ethical considerations ........................................................................................................................ 56. CHAPTER 4 - THE ALOA CONCEPTUAL MODEL................................................................. 57 4.1. The rBloom matrix ............................................................................................................................. 58. 4.2. Assessment methods ......................................................................................................................... 67. 4.3. E-assessment tasks............................................................................................................................. 86. 4.4. The description of LO in EE ................................................................................................................. 92. 4.5. Defining relations ............................................................................................................................. 107. CHAPTER 5 - IMPLEMENTATION ........................................................................................ 113 5.1. Implementation scenarios................................................................................................................ 113. 5.2. Practical tools .................................................................................................................................. 118. 5.3. Case studies ..................................................................................................................................... 123. CHAPTER 6 - INTERPRETATION OF RESULTS ................................................................ 135 6.1. Application of the ALOA model ........................................................................................................ 135. 6.2. Adequacy of the chosen LOs in EE (RQ1) .......................................................................................... 136. 6.3. Adequacy of the selection of assessment methods (RQ2) ................................................................ 137. 6.4. Adequacy of the ALOA model to describe assessment ..................................................................... 137. VIII.

(9) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. 6.5. The alignment question (RQ3 and RQ4) ...........................................................................................138. CHAPTER 7 - CONCLUSIONS .................................................................................................. 141 7.1. Conclusions regarding the research problem....................................................................................141. 7.2. Theoretical implications of the research project ..............................................................................143. 7.3. Practical implications of the research project...................................................................................144. 7.4. Implications for future research .......................................................................................................146. 7.5. Final remarks....................................................................................................................................148. REFERENCES .................................................................................................................................. 149 ANNEXES ............................................................................................................................................... I Annex I - DATABASE DIAGRAMS ................................................................................................ III Annex II - TEMPLATES FOR THE DIFFERENT TYPES OF CASE STUDIES ......................... V Annex III - CASE STUDY 1 ............................................................................................................. VII Annex IV - CASE STUDY 2 .............................................................................................................. IX Annex V - CASE STUDY 3 .............................................................................................................. XI Annex VI - CASE STUDY 4 ........................................................................................................... XIII. IX.

(10) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. X.

(11) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. ABSTRACT This study researched the question of alignment between the intended learning outcomes and assessment in the field of engineering education. Based on the initial problem, four research questions were developed that focused on: defining and describing learning outcomes in engineering; defining and describing assessment methods and e-assessment tasks; defining a model for achieving alignment between learning outcomes and assessment; finally, applying the model to link specific learning outcomes with specific assessment methods. During this study a conceptual model was developed with the goal of providing an answer to the research problem, the ALOA (Aligning Learning Outcomes with Assessment). The model was derived from literature research and included different components directly linked with the research questions. The first one is a selection and description of learning outcomes in the field of engineering that was based on existing qualification frameworks of the sector: ABET and EUR-ACE. The second component was a selection and description of assessment methods. This selection was derived from literature and compiled in a structured list that included six general assessment categories that are then divided in specific assessment methods. Both the learning outcomes and the assessment methods were described using an adapted version of the Taxonomy Table developed by Anderson et al. in their work A Taxonomy for Learning, Teaching and Assessing. The Taxonomy Table describes the assessment items and the learning outcomes using a two dimensional classification system based on knowledge and cognitive processes. During the development of the ALOA model each selected learning outcome and specific assessment method were analysed and classified in terms of type of knowledge and cognitive processes addressed. This detailed description was used for answering the main question of the research problem. By using the same classification system for both types of items, it was possible to develop an alignment proposal. The ALOA conceptual model was developed with the intention of being used by different stakeholders in Higher Education Institutions. The study defined four different scenarios of implementation of the ALOA model: to verify current alignment in existing courses; to develop an assessment strategy based on statements of learning outcomes; to verify vertical alignment of courses with higher level learning outcomes; to verify horizontal alignment of learning outcomes defined at the same level but in different contexts, as in mobility situations. The ALOA model was translated in practical tools and tested for applicability using a multiple case study approach. The results of the case studies are mainly concerned with the improvement of the model by detecting problems and suggesting changes. The main theoretical findings of the study were related with the clarification of core concepts in terms of assessment methods and assessment tasks. There was also an attempt to structure general information concerning assessment and e-assessment. In terms of the concept of alignment this study contributes with an innovative perspective that includes the clarification of the meaning of alignment and the definition of four criteria for achieving alignment: match, emphasis, coverage and precision.. XI.

(12) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. XII.

(13) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. RESUMO Este estudo investigou a questão de alinhamento entre os resultados de aprendizagem e a avaliação, na área da educação da engenharia. A partir do problema inicial forma definidas quarto questões a investigar: como definir e descrever os resultados de aprendizagem em engenharia; como selecionar e descrever métodos de avaliação e métodos de e-assessment; como definir um modelo que permita atingir o alinhamento entre os resultados de aprendizagem e os métodos de avaliação; como aplicar o modelo de forma a obter a ligação entre certos tipos de resultados de aprendizagem e métodos de avaliação específicos. Durante este estudo for desenvolvido um modelo conceptual que tem como objectivo dar resposta ao problema inicial, o modelo ALOA (Aligning Learning Outcomes with Assessment). O modelo foi obtido a partir de investigação de literatura e inclui componentes que abordam cada uma das questões acima mencionadas. A primeira componente é uma seleção e descrição dos resultados de aprendizagem na área da engenharia que foi obtida a partir de quadros de qualificações existentes no sector: ABET e EUR-ACE. A segunda componente é uma seleção e descrição de métodos de avaliação que foi obtida a partir da análise de obras de referência na área. A seleção dos métodos de avaliação foi organizada numa lista com seis categorias gerais em que cada uma inclui vários métodos específicos. Tanto os resultados de aprendizagem como os métodos de avaliação foram descritos recorrendo a uma versão adaptada da Tabela Taxonómica desenvolvida por Anderson et al. na obra A Taxonomy for Learning, Teaching and Assessing. A Tabela Taxonómica define um sistema de classificação com duas dimensões: tipo de conhecimento e processos cognitivos. Durante o desenvolvimento do modelo ALOA cada resultado de aprendizagem e cada método de avaliação foi analisado e classificado em termos de cada uma destas dimensões. Esta descrição detalhada dos itens foi utilizada para responder à questão principal deste projeto de investigação que era o alinhamento entre os resultados de aprendizagem e a avaliação. Ao utilizar o mesmo de sistema de classificação para os dois tipos de itens foi possível desenvolver uma proposta de alinhamento. O modelo conceptual ALOA foi desenvolvido com o objective de ser usado por diferentes tipos de utilizadores em instituições de ensino superior. O estudo definiu quarto cenários de implementação para o modelo: verificação do alinhamento em unidades curriculares existentes; definição de uma nova estratégia de avaliação tendo como base os resultados de aprendizagem pretendidos; verificação do alinhamento vertical de uma unidade curricular por comparação com resultados de aprendizagem definidos a um nível superior; verificação do alinhamento horizontal de duas unidades curriculares definidas ao mesmo nível mas em contextos diferentes como ocorre em situações de mobilidade. O modelo ALOA foi traduzido em ferramentas práticas e a sua aplicabilidade foi testada recorrendo a múltiplos casos de estudo. As principais conclusões teóricas deste estudo estão relacionadas com a clarificação dos conceitos métodos de avaliação e práticas de avaliação. Houve também uma tentativa da qual resultou uma proposta de estruturação de informação relativa à avaliação e e-assessment. Em relação ao conceito de alinhamento, este estudo contribui com uma perspectiva inovadora que inclui a clarificação do significado do termo alinhamento e a definição de quarto critérios para que se possa atingir o alinhamento: coincidência, ênfase, cobertura e precisão.. XIII.

(14) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. XIV.

(15) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. RESUME Cette étude a examiné la question de l'alignement entre les acquis d'apprentissage et d'évaluation dans le domaine de la formation des ingénieurs. D’aprés le problème initial ont défini les quatres questions de la recherche: comment définir et décrire les résultats d'apprentissage dans l'ingénierie;comment choisir et decrireles méthodes d'évaluation et les méthodes d'évaluation électronique; comment définir un modèle qui permettra d'atteindre l'alignement entre les résultats de l'apprentissage et les méthodes d'évaluation; comment 'appliquer le modèle de façon à obtenir la liaison entre certains types de résultats d'apprentissage et les méthodes d'évaluation spécifiques. Au cours de cette étude, on a developpé un modèle conceptuel qui vise à répondre au problème de départ , le modèle ALOA (Aligning Learning Outcomes with Assessment). Le modèle a été dérivée à partir de la littérature scientifique et comprend des composantes qui répondent à chacune des questions mentionnées ci-dessus. La première composante est une sélection et une description des résultats d'apprentissage en ingénierie qui a été obtenue à partir des cadres de qualifications du secteur: ABET et EUR-ACE. La deuxième composante est une sélection et une description des méthodologies qui ont été obtenus à partir de l'analyse des ouvrages de référence dans le domaine. Le choix des méthodes d'évaluation a été organisée dans une liste avec six grandes catégories dans lesquelles s’incluent les plusieurs méthodes spécifiques. Les résultats d'apprentissage et les méthodes d'évaluation ont été décrites à l'aide d'une version adaptée de la taxonomie développé par Anderson et al. Dans le travail sur la taxonomie de l'apprentissage, l'enseignement et l'évaluation. Cette taxonomie tdéfine un système de classification à deux dimensions: type de de connaissance et processus cognitif. Au cours du développement du modèle ALOA on a été evalué chaque résultat de l'apprentissage et chaque méthode d'évaluation en chacune de ces dimensions. Cette description détaillée des éléments a été utilisé pour répondre à la question principale de ce projet de recherche qu’était l'alignement entre les résultats d'apprentissage et d'évaluation. En utilisant le même système de notation pour les deux types d'articles a été possible développer une proposition d’alignement. Le modèle conceptual ALOA a été élaboré pour être utilisé par différents utilisateurs dans les établissements d'enseignement supérieur. L'étude a défine quatre scénarios de déploiement pour le modèle: vérification de l'alignement dans les cours existants; définition d'une nouvelle stratégie d'évaluation basée sur les résultats d'apprentissage escomptés; vérification de l'alignement vertical d'une unité d'enseignement par rapport aux résultats d'apprentissage définis à un niveau supérieur; vérification de l'alignement horizontal des deux cours fixés au même niveau, mais dans des contextes différents en situation de mobilité. Le modèle ALOA a été traduit en outils pratiques et leur applicabilité a été testée à l'aide de plusieurs 'études de cas . Les principales conclusions théoriques de cette étude sont liées à clarifier les concepts, les méthodes et les pratiques d'évaluation. Il y avait aussi une tentative qui a mené à une proposition de structuration de l'information sur l'évaluation et l'e-évaluation. En ce qui concerne le concept de l'alignement, cette étude fournit une nouvelle perspective qui inclut la clarification de la signification du terme et la définition de quatrecritère afin que nous puissions parvenir à un alignement: coïncidence, l'accent, la couverture et la précision.. XV.

(16) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. XVI.

(17) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. LIST OF TERMS AND ACRONYMS IN USE ABET. Organization that accredits college and university programs in the fields of applied sciences and engineering. Formerly known as Accreditation Board for Engineering and Technology. ALOA. Model for Aligning Learning Outcomes with e-Assessment. AT. Assessment task. CDIO. Conceiving — Designing — Implementing — Operating real-world systems and products. CE. Continuing education. CEI. Continuing education institutions. EE. Engineering education. EUR-ACE. European quality label for engineering degree programmes. HE. Higher education. HEI. Higher education institutions. ICT. Information and communication technologies. iLO. Intended learning outcome. LO. Learning outcome. LT. Learning technologies. MCQ. Multiple choice question. PL. Prior learning. rBloom. Adapted version of the revised Bloom’s taxonomy. RPL. Recognition of prior learning. RQ. Research question. SAQ. Short answer question. TLA. Teaching and Learning activities. TT. Taxonomy Table. XVII.

(18) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. XVIII.

(19) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. LIST OF TABLES TABLE 1 - EXAMPLES OF SEARCH EXPRESSIONS USED ............................................................................................ 10 TABLE 2 - TYPES OF SOURCES USED AND EXAMPLES .............................................................................................. 11 TABLE 3 - TRENDS IN ASSESSMENT. FROM BROWN ET AL. [7, P.13] ....................................................................... 13 TABLE 4 – PRINCIPLES AND PRACTICES ON ASSESSMENT (EXTRACTED AND ADAPTED FROM WOODS [50]) ........... 16 TABLE 5 - REASONS FOR ASSESSING STUDENTS ...................................................................................................... 16 TABLE 6 - SUMMARY OF ASSESSMENT METHODS ................................................................................................... 22 TABLE 7 - FAMILIES OF ESSAY QUESTIONS ............................................................................................................. 24 TABLE 8 - GENERAL CRITERIA FOR ASSESSING ESSAYS (ADAPTED FROM BROWN ET AL. [7]) ................................. 25 TABLE 9 - PROBLEM SOLVING STAGES AND COGNITIVE PROCESSES (ADAPTED FROM WOODS [55, P.448]) ............ 26 TABLE 10 - SUMMARY OF PROBLEM-SOLVING AS VIEWED BY PLANTS ET AL. [56] ................................................. 26 TABLE 11 - SUMMARY OF PRACTICAL ENQUIRY AS VIEWED BY HERRON ............................................................... 27 TABLE 12 - ASSESSING EXPERIMENTAL PROJECT (ADAPTED FROM BROWN ET AL. [7, P.128-9]) ............................ 28 TABLE 13 - BRIEF ANALYSIS OF CRISP’S E-ASSESSMENT ITEMS.............................................................................. 32 TABLE 14 - DIFFERENCES BETWEEN SPECIFICITY LEVELS OF OBJECTIVES (FROM ANDERSON ET AL. [40, P.17]) .... 35 TABLE 15 - SUMMARY OF BLOOM'S TAXONOMY [41] ............................................................................................ 36 TABLE 16 - SUMMARY OF THE KNOWLEDGE DIMENSION OF RBLOOM .................................................................... 37 TABLE 17 - SUMMARY OF THE COGNITIVE DIMENSION ........................................................................................... 38 TABLE 18 - AN EXAMPLE OF A TAXONOMY TABLE USED TO VERIFY ALIGNMENT .................................................. 40 TABLE 19 - COMPARISON BETWEEN QUALIFICATIONS IN EE .................................................................................. 44 TABLE 20 - SUMMARY OF DATA COLLECTION ........................................................................................................ 53 TABLE 21 - SUMMARY OF THE ANALYSIS OF DATA ................................................................................................. 54 TABLE 22 - THE TAXONOMY TABLE BY ANDERSON ET AL ..................................................................................... 59 TABLE 23 - FACTUAL KNOWLEDGE (EXTRACTED FROM ANDERSON ET AL. [40, P. 29]) .......................................... 60 TABLE 24 - CONCEPTUAL KNOWLEDGE (EXTRACTED FROM ANDERSON ET AL. [40, P. 29]).................................... 61 TABLE 25 - PROCEDURAL KNOWLEDGE (EXTRACTED FROM ANDERSON ET AL. [40, P. 29]).................................... 61 TABLE 26 – METACOGNITIVE KNOWLEDGE (EXTRACTED FROM ANDERSON ET AL. [40, P. 29]) ............................. 62 TABLE 27 - REMEMBER CATEGORY (ADAPTED FROM ANDERSON ET AL. [40, P. 67/8]) ........................................... 63 TABLE 28 - UNDERSTAND CATEGORY (ADAPTED FROM ANDERSON ET AL. [40, P. 67/8]) ....................................... 63 TABLE 29 – APPLY CATEGORY (ADAPTED FROM ANDERSON ET AL. [40, P. 67/8]) .................................................. 64 TABLE 30 - ANALYSE CATEGORY (ADAPTED FROM ANDERSON ET AL. [40, P. 67/8]) .............................................. 65 TABLE 31 - EVALUATE CATEGORY (ADAPTED FROM ANDERSON ET AL. [40, P. 67/8]) ............................................ 65 TABLE 32 – CREATE CATEGORY (ADAPTED FROM ANDERSON ET AL. [40, P. 67/68]).............................................. 66 TABLE 33 – RBLOOM MATRIX, AN ADAPTED VERSION OF THE TAXONOMY TABLE BY ANDERSON ET AL ............... 67 TABLE 34 - MAPPING OF ASSESSMENT BASED ON MCQ TO BLOOM'S REVISED TAXONOMY ................................... 69 TABLE 35 - MAPPING OF ASSESSMENT BASED ON ESSAYS TO BLOOM'S REVISED TAXONOMY ................................ 73 TABLE 36 - MAPPING OF SIMPLE CLOSED-ENDED PROBLEM SOLVING TO BLOOM'S REVISED TAXONOMY (DIAGNOSIS + ROUTINE ACTIVITIES) .................................................................................................................................. 76. XIX.

(20) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. TABLE 37 - MAPPING OF COMPLEX CLOSED-ENDED PROBLEM SOLVING TO BLOOM'S REVISED TAXONOMY ........... 76 TABLE 38 - MAPPING OF OPEN ENDED PROBLEM SOLVING TO BLOOM'S REVISED TAXONOMY ................................ 77 TABLE 39 - MAPPING OF PRACTICAL WORK TO BLOOM'S REVISED TAXONOMY ...................................................... 79 TABLE 40 - MAPPING OF ASSESSMENT USING SHORT-ANSWER QUESTIONS TO BLOOM'S REVISED TAXONOMY ....... 83 TABLE 41 - MAPPING OF REFLECTIVE PRACTICE ASSESSMENT TO BLOOM'S REVISED TAXONOMY .......................... 85 TABLE 42 - COMPLETE MAPPING OF ABET PROGRAMME OUTCOMES TO BLOOM'S REVISED TAXONOMY. ............ 100 TABLE 43 - COMPLETE MAPPING OF EUR-ACE PROGRAMME OUTCOMES TO BLOOM'S REVISED TAXONOMY. ..... 107 TABLE 44 – MAPPING OF THE ABILITY TO USE NEWTON’S FIRST LAW IN ROUTINE PROBLEM SOLVING. ................ 109 TABLE 45 - STAKEHOLDERS OF DIFFERENT SCENARIOS FOR ALIGNMENT.............................................................. 118 TABLE 46 – NUMBERING OF THE CELLS TO TRANSFORM TWO DIMENSIONS INTO ONE........................................... 120 TABLE 47 - LINK BETWEEN ILOS AND THE LOS OF ABET ................................................................................... 126 TABLE 48 - LINK BETWEEN ILOS AND THE LOS OF EUR-ACE ............................................................................ 126 TABLE 49 - SUMMARY OF METHODS OF ASSESSMENT USED IN THE CASE STUDY .................................................. 127 TABLE 50 - OVERLAPPING OF RBLOOM FOR SIMPLE PROBLEM SOLVING WITH ITEM CS.02.A03.14A ................... 127 TABLE 51 - OVERLAPPING OF RBLOOM FOR OPEN PROBLEM SOLVING WITH ITEM CS.02.A03.14B....................... 127 TABLE 52 - OVERLAPPING OF RBLOOM FOR PROJECT WITH ITEM CS.02.A02 ....................................................... 128 TABLE 53 - ALIGNMENT OF LO1 WITH REAL ASSESSMENT ................................................................................... 129 TABLE 54 - ALIGNMENT OF LO2 WITH REAL ASSESSMENT ................................................................................... 129 TABLE 55 - ALIGNMENT OF LO3 WITH REAL ASSESSMENT ................................................................................... 129 TABLE 56 - ALIGNMENT OF LO4 WITH REAL ASSESSMENT ................................................................................... 130 TABLE 57 - ALIGNMENT OF LO1 WITH STANDARD ASSESSMENT .......................................................................... 130 TABLE 58 - ALIGNMENT OF LO2 WITH STANDARD ASSESSMENT .......................................................................... 130 TABLE 59 - ALIGNMENT OF LO3 WITH STANDARD ASSESSMENT .......................................................................... 130 TABLE 60 - ALIGNMENT OF LO4 WITH STANDARD ASSESSMENT .......................................................................... 131 TABLE 61 - MATCHING SCORES OBTAINED FOR THE EACH ILOS WHEN COMPARED WITH STANDARD ASSESSMENT METHODS ...................................................................................................................................................... 131. TABLE 62 - OVERLAPPING OF STANDARD ALOA SIMPLE PROBLEM SOLVING MATRIX WITH REAL ASSESSMENT FROM CASE-STUDY ....................................................................................................................................... 132. TABLE 63 - OVERLAPPING OF STANDARD ALOA OPEN PROBLEM SOLVING MATRIX WITH REAL ASSESSMENT FROM CASE-STUDY ................................................................................................................................................. 133. TABLE 64 - OVERLAPPING OF STANDARD ALOA PROJECT MATRIX WITH REAL ASSESSMENT FROM CASE-STUDY 133 TABLE 65 - RELATION BETWEEN THE PROBLEM AND THE ALOA MODEL ............................................................. 142. XX.

(21) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. LIST OF FIGURES FIGURE 1 - FIELD OF STUDY...................................................................................................................................... 2 FIGURE 2 - ASSESSMENT CYCLE AS DESCRIBED BY BROWN ET AL .......................................................................... 15 FIGURE 3 - PURPOSES OF ASSESSMENT ................................................................................................................... 17 FIGURE 4 - SOURCES OF ASSESSMENT .................................................................................................................... 18 FIGURE 5 – CURRENT CHALLENGES IN ASSESSMENT .............................................................................................. 18 FIGURE 6 - DIFFERENT DIMENSIONS OF RELIABILITY OF ASSESSMENT .................................................................... 19 FIGURE 7 - TYPES OF VALIDITY RELATED WITH CURRENT RESEARCH ..................................................................... 20 FIGURE 8 - THE LEARNING CYCLE AS VIEWED BY KOLB ......................................................................................... 29 FIGURE 9 - GROUPS OF ACTIVITIES RELATED WITH E-ASSESSMENT IMPLEMENTATION [26] .................................... 30 FIGURE 10 - THE SAME LO SHOULD BE PRESENT IN ALL THE ACTIVITIES ............................................................... 41 FIGURE 11 - RELATION BETWEEN THE CONCEPTUAL MODEL AND THE RESEARCH QUESTIONS................................ 49 FIGURE 12 – INITIAL REPRESENTATION OF THE PROBLEM....................................................................................... 49 FIGURE 13 - SECOND VERSION OF THE MODEL........................................................................................................ 50 FIGURE 14 - THIRD VERSION OF THE MODEL........................................................................................................... 51 FIGURE 15 - FOURTH AND FINAL VERSION OF THE MODEL ...................................................................................... 52 FIGURE 16 - THE ALOA CONCEPTUAL MODEL ....................................................................................................... 57 FIGURE 17 - ALIGNMENT POSSIBILITIES FOR ONE UNIT OR COURSE ...................................................................... 107. XXI.

(22) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. XXII.

(23) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. CHAPTER 1 - INTRODUCTION. 1.1. BACKGROUND. Since the last half of the 20th Century, the World has been experiencing rapid transformation in the field of Education led by the changing Knowledge Society [1]. As Peter Drucker explained in 1996, in this new society access to work is only gained through formal education and not acquired through apprenticeship. Almost two decades have passed and this is already what is happening in some parts of the World. Education and schooling have become a major concern for the society and it is a priority in national and transnational policies. The strategic document Europe 2020 [2] defines as a priority the development of an economy based on knowledge and innovation. All educational institutions play an important role in achieving this goal. Higher Education (HE), Continuing Education (CE) and Vocational Training have been most affected by this transformation, adapting to the demand for new skills of the labour market and at the same time corresponding to the needs of an increasing number of students [3]. The global economy created opportunity and need for the mobility of students and workers, demanding efficient recognition of qualifications and increasing competitiveness in this field. The labour market demands more workers qualified and updated and this trend is mirrored in educational policies in Europe [4-6]. All this generates pressure towards a quality-based approach for all education providers, as Drucker predicted in his article about the knowledge society[1]. One visible effect of this transformation is the shift from a content-based approach in Education to an approach centred on the student and what he/she has learned and achieved. This transformation has led to different trends in HE as identified by Brown et al. [7]. One of the trends is the change from structuring education based on what should be taught by defining educational objectives to structuring education by defining what students should learn, the Learning Outcomes (LOs). This approach is underpinning the development and implementation of most European Education policies at international and national levels [8-10]. In Europe, HEI and CEI are redefining programmes in terms of Learning Outcomes, harmonizing them with national, international and sector level frameworks of qualifications that are also based on Learning Outcomes [11-14]. Several projects and initiatives are. 1.

(24) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. working towards the definition LOs, specific and transversal that can be used as a common reference [15, 16]. Learning Outcomes are also becoming fundamental for structuring the standards and guidelines of quality assessment of HE and CE institutions in Europe and worldwide [17-20]. The field of engineering education has also been affected by these current trends. Learning outcomes are being used in qualification frameworks specific of engineering, like ABET, EUR-ACE and CDIO. In this context, the assessment of Learning Outcomes becomes a crucial process for the Educational System. It should be a major concern of educational institutions to ensure that assessment of student learning is being guided by what they should be learning, i.e. assessment should be aligned with the intended learning outcomes (iLOs). Another major revolution in our society has been the introduction of Information and Communication Technologies. The use of ICT applied to Education, e-learning, has been increasing and its use creates new opportunities for teaching, learning and assessment and has huge potential as an answer to some of the current challenges of Education. The change to the digital media has impact on the availability, reusability, accessibility and cost of learning resources, complemented by the communication and networking potential of the Internet that takes Education to a Global level [21] [22, 23]. The application of ICT to education and in particular to assessment is a subject of great discussion. Some of the issues related with the use of e-learning in assessment are related with validity and reliability of the process. This study will be focusing on this particular issue. 1.2. FIELD OF STUDY AND INTENDED STAKEHOLDERS. The current study will be looking into the relationship between learning and assessment. The focus of the study is placed in the intersection of three areas, as illustrated by Figure 1: learning outcomes, assessment of student’s learning and e-learning. It will be focusing on Higher Education but more specifically on Engineering Education. The question of alignment of e-assessment and education is represented by the area where the three circles overlap.. Learning Outcomes. Assessment. e-Learning. Figure 1 - Field of study. The wide use of e-learning is already impacting Education, promoting change and innovation in different aspects including pedagogy, technology, organization, accessibility, and flexibility among others [24]. It is a complex and multidisciplinary area and, given its impact, it is important that Elearning research be informed by evidence [23]. Current literature reviews in this area indicate that elearning approaches to assessment lack the pedagogical framework and most research describes. 2.

(25) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. implementation studies at course level [25]. The present research intends to contribute to establishing a conceptual model for the implementation of e-assessment in Engineering Education. Assessment is a crucial process of Education and is seen by current trends as part of the learning process and not as a separate event [7]. Assessment of student learning will encourage involvement of students and provides feed-back to students and teachers [26]. It has an important role in validation and certification and is deeply related with quality issues as will be explained below. The current study will be focusing on the specific link between what should be learned and what should be assessed. The purpose of this study is to research the applicability of e-assessment strategies to the field of engineering education. It intends to help in the definition of adequate e-assessment strategies based on what is the stated intended learning. This research study mainly applies to engineering courses that use or intend to use e-assessment strategies. The implementation stage addressed existing courses of Faculdade de Engenharia da Universidade do Porto (FEUP). It focused on the stated learning outcomes of the courses and on the existing assessment and e-assessment strategies as important components of the alignment question. As the study will focus at course level, it will be strongly related with the activities of faculty that are usually the ones that define the iLOs and the assessment tasks. However, the results of the study have impact on other levels, including programme and qualification frameworks. It is expected that other stakeholders including curriculum designers, mobility staff, and quality and accreditation staff will be using the results of this study. In terms of students, they are not considered direct users of the study and tools. Still, the project is based on the learner centred perspective of education and they will benefit from a more efficient and effective educational process. 1.3. BACKGROUND AND MOTIVATION OF THE RESEARCHER. Since 1998 I have been working in the field of ICT applied to education. I started working as an editor of multimedia projects in an educational publishing company. Later in 2001, I continued the work in this field as responsible for the e-learning unit of Universidade do Porto. This position gave me the opportunity to contribute to the implementation of an institutional strategy for e-learning. At the same time, it allowed me to build a transversal perspective of the implementation of learning technologies at the institutional course level, including different learning management systems (LMS) and other learning technologies (LTs). This role included giving individual support to teachers in terms of pedagogical and technological issues. It was possible to learn from the different strategies used by the teachers in terms of teaching and assessment and to perceive the perspective of the students. Additionally it included the active participation in European research projects in the field of e-learning which was important to learn and update knowledge and skills in this area, During this period I was a teacher in traditional courses at the university. As I developed blended learning strategies to work with the students I could better understand the perspective of the teachers and learners when dealing with e-learning. I was involved in the development, teaching and tutoring of distance e-learning courses that contributed to improve my knowledge in this area. Finally, I was a student in a Master’s programme of the School of Engineering that included an e-learning course with e-assessment tasks. This gave me the perception of the student view in terms of e-learning. This summary of activities shows that the motivation for the field of study comes from both the academic and the professional area. When dealing with e-learning strategies from each of the perspectives, assessment has always been a component that raises questions and problems. The flexibility of the e-learning environment creates opportunities for experimenting and many interesting. 3.

(26) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. and creative examples of e-assessment tasks can be found at Universidade do Porto. A recent trend at the institution showed that teachers are using e-learning based assessment for summative purposes. Most of the e-assessment implementations were online exams but the teachers successfully developed other types of tasks [27-29]. All of these represent a very interesting field of research. The application to the field of engineering comes basically from the same background. From the researcher perspective, there are two subject areas that are quite innovative in the use of Learning Technologies: medicine and engineering. Also, both areas have strong civil responsibility and the need to have regulations from professional bodies. Both aspects have impact on the mobility of the students and workers. The affiliation to the School of Engineering as a student and a former invited teacher both contributed to my decision of the scope of application of the research. Finally, I worked closely in the creation and development of the European project VIRQUAL that was related with field of qualification frameworks, learning outcomes and assessment in the context of international mobility [30-33] which developed an interest in the area of transnational Education policies, Qualification Frameworks and Quality Assurance in HE, a field that has gained an increased importance in Higher Education. The combination of these different areas of interest contributed to the definition of the problem and to the research approach that came after the initial research. 1.4. STATEMENT OF THE PROBLEM AND RESEARCH QUESTIONS. The research problem is placed in the area of interaction of three fields of study: Assessment, Learning Outcomes and e-Learning. The scope of application will be engineering education. Engineering education has specificities and requisites [34] that may constitute challenges in the implementation of assessment based in Learning Technologies (LT). This study intends to contribute to developing elearning strategies for assessing EE, helping to deal with current challenges in education. In general terms, the approach chosen was to develop a model that matches specific e-assessment methods to specific LOs in the field of engineering. This means that it might be possible for teachers to define the intended Learning Outcomes (iLO) of an online course and from this definition to have an indication of the e-assessment methods they might consider using. Formally, this problem is defined as: To what extent may e-assessment methods be used to measure the achievement of Learning Outcomes in engineering education?. Given this problem, it was necessary to recognize that there are a wide variety of engineering schools, engineering programmes and engineering courses. There are different qualification frameworks that use LOs in the engineering sector. So, the first challenge was how to select the LOs that were going to be used for the purpose of the research. The selection process had to take in consideration the subject area, level and even the nature of the LOs. The same problem existed in relation to the assessment methods. Assessment tasks are usually defined at course level, even though some examples can be found at a higher level. Again, there are a considerable variety of assessment and e-assessment tasks that could be used in this study. Another aspect of assessment is that in most cases is deeply embedded in the structure of the course or unit and highly contextualized.. 4.

(27) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. The first two research questions translate this need for definition of the central concepts of the problem: RQ1) Which type of Learning Outcomes in the field of Engineering are relevant and should be considered? RQ2) What are the e-assessment methods that should be considered?. After defining the concepts, it is necessary to deal with the core of the problem that is the relationship between assessment and LOs. This link between the iLOs and the assessment tasks is part of what is considered the alignment of a course [35, 36]. The first concern with alignment is if every iLO is assessable. It is a recognized issue in HE the need to include transversal skills in curricula and how difficult it is to assess them. In engineering, curricula are pressured to include these types of LOs among the technical LOs specific of the sector. This is a known challenge in EE [34] that may also represent a challenge to effective assessment. The last two research questions of this study are related with the alignment question: RQ3) What type of intended Learning Outcomes can be measured by eassessment methods? RQ4) Is it possible to propose specific e-assessment strategies for each type of LO in EE?. 1.5. IMPACT EXPECTATIONS OF THE STUDY. Assessment is an important issue in HE that has high impact on learning [3, 7, 37, 38]. It raises questions about the efficiency, effectiveness and adequacy of different methods and strategies. Eassessment, due to the technological component and the association with distance learning brings additional controversies, some of them intrinsic to e-learning. As described by Conole and Martin, elearning is a complex area that may have impact in a diversity of fields including the teaching and learning process (T/L), organizational structures, political and socio-cultural issues, among others [22]. It is expected that the findings of this research project will: •. • •. •. •. Help teachers to decide which assessment tasks are more suitable for the LOs stated for a specific course or module. The ALOA model developed includes practical tools that will support the decision process. Help teacher and other stakeholders to verify the current alignment of existing courses. This is one of the scenarios of application of the ALOA model. Contribute to develop a pedagogical framework for the implementation of aligned eassessment strategies, based on the definition of Learning Outcomes. The ALOA model was developed as a conceptual and theoretical model that intends to promote the alignment between iLOs and assessment. Facilitate accreditation processes and navigation between different qualification frameworks, by providing a common tool for the description of LOs. This was defined in the current study as vertical alignment and is one of the implementation scenarios of the ALOA model. Promote mobility and recognition of prior learning, formal or informal by allowing the comparison of the LOs of previous experiences with the intended ones.. 5.

(28) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. 1.6. STRUCTURE OF THE STUDY. This dissertation is divided in seven chapters. This chapter defines the context of the current research study and introduces the main thematic areas that will be explored during the development of the work. After providing a rationale for the work, the chapter will present the background of the researcher and the motivations for conducting this study. Following, the research problem is described as well as the research questions that guided the project. Finally, the chapter addresses the expected impact of the research and includes a description of the structure of this dissertation, with a brief explanation of the contents of each chapter. Chapter 2 provides a theoretical background for the problem and a critical analysis of the published work in the field. From the analysis of theory and published work it was possible to define the approach that was followed to address the problem. The ALOA conceptual model was derived from the literature research so this chapter grounds most the decisions that were taken during development of the work. Chapter 3 describes the different components of the research method, including the theoretical development of the conceptual model and the multiple case studies approach used in the implementation stage. The chapter also explains how the research project dealt with validity, reliability and ethical issues. In Chapter 4 the conceptual model is described in detail. It includes the description of the main concepts of the model: LOs in engineering and assessment methods. The chapter also includes a definition of the relationships between the concepts that are included in the ALOA model. Chapter 5 describes the implementation of the model. It starts by identifying and describing some implementation scenarios and the practical tools that were developed. Finally, the chapter describes the implementation of the ALOA model to the case studies that were used to test the applicability of the model. In Chapter 6 the results of the applicability test are interpreted from the perspective of each case and from a transversal point of view. In this chapter, the results of the implementation stage are analysed from the perspective of the research questions. Chapter 7 is the final chapter of the dissertation. It starts by presenting the global conclusions of the work from the perspective of the research problem. Then, the results are analysed in terms of the contributions to theory and practice. Finaly, it explores future work that could be developed based on the current project. 1.7. PUBLICATIONS RELATED WITH THE RESEARCH STUDY •. E-ASSESSMENT OF LEARNING OUTCOMES IN ENGINEERING EDUCATION”, Falcão, R., Proceeding of WEEF 2012, Buenos Aires, Argentina, October 2012 (accepted). •. “A CONCEPTUAL MODEL FOR E-ASSESSING STUDENT LEARNING IN ENGINEERING EDUCATION”, Rita Falcão, Proceedings of ICL2012, Villach, Switzerland, September 2012 (accepted). •. “A CONCEPTUAL MODEL FOR E-ASSESSING STUDENT LEARNING IN ENGINEERING EDUCATION “, Falcão, R, SITE 2012, Austin, Texas, USA, March 2012. •. “ASSESSING LEARNING OUTCOMES IN ENGINEERING EDUCATION: AN ELEARNING BASED APPROACH”, WEE 2011, Lisboa, Portugal, September 2011. 6.

(29) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. •. “Assessment of Learning Outcomes through e-Learning”, WCCEE2010, Singapore, October 2010. •. "MEASURING IMPACT OF E-LEARNING ON QUALITY AND LEARNING OUTCOMES – A PROPOSAL FOR A PHD PROJECT", Falcão, R., Soeiro, A., EDEN 2007, Junho 2007;. 7.

(30) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. 8.

(31) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. CHAPTER 2 - RELATED WORK. 2.1. INTRODUCTION. The field of study of this dissertation is defined by the intersection of three areas: assessment, learning outcomes and e-learning. Given this field, the research focused in the area of Higher Education and in particular Engineering Education. Given the general field of the research, what was expected to obtain from the literature review? What guided the literature research was the analysis of the problem and of the research questions. To what extent e-assessment methods may be used to measure the achievement of intended Learning Outcomes in engineering education?. The problem is composed by two dimensions and intends to explore a specific relation between them. The dimensions “Learning Objects in Engineering Education” and “e-assessment methods” are the focus of the first two research questions. RQ1) Which Learning Outcomes in the field of Engineering are relevant and should be considered? RQ2) Which are the on-line assessment methods that should be considered?. What is intended with these two questions is a clarification of the dimensions that will be analysed so it is possible to obtain a valid selection or at least an informed selection process. Literature research was fundamental to analyse this aspect of the problem. The first dimension to be analysed was the LOs in EE. Starting with the general approach, it was necessary to clarify certain aspects related with LOs including terminology, definitions and the theory behind them. Concerning EE, it was researched what was the role of LOs in EE, how were they implemented and if there was a trend for a common set of LOs in engineering courses. Concerning the second dimension, e-assessment, it was possible to find a wide variety of published work with a corresponding variety of assessment methods. However, most of the published work was. 9.

(32) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. based on individual cases of application. It was difficult to find fundamentals of e-assessment or a comprehensive study of e-assessment methods. Early in the literature research of this subject it was considered necessary to go back to the basics of assessment. Donnan [39] faced a similar situation in his work. Educational developers considered that e-assessment or online assessment was not distinct from general assessment. So the analysis again started from the perspective of general assessment to clarify several issues and finally specific e-assessment methods were researched. The third component of the problem is a specific relation between the dimensions, the alignment between LOs in EE and e-assessment methods. That is the core of the problem and the focus of research questions 3 and 4. RQ3) What type of intended Learning Outcomes can be measured by eassessment methods? RQ4) Is it possible to propose specific e-assessment strategies for each type of LO in EE?. In this stage, literature research was centred on finding suggestions on using specific assessment methods to measure or evaluate specific types of Learning Outcomes, in general and in the field of Engineering. An important finding in terms of related work, was the book “A Taxonomy for Learning, Teaching and Assessment” [40] that became a central piece of the this research project.. 2.1.1. LITERATURE RESEARCH METHOD. The initial stage of the literature research was exploratory with a view of obtaining a framework for the research problem. The more systematic approach was guided by the analysis of the problem and previous knowledge on the subject. Table 1 summarizes the main search expressions used. Table 1 - Examples of search expressions used Research question RQ1. RQ2. RQ3. RQ4. Research terms Learning Outcomes or Learning Objective and Engineering, Higher Education, qualification frameworks, ABET, EUR-ACE assessment or e-assessment or evaluation and methods, tools, online, CBT (computer based testing), CBA (computer based assessment), CAA (computer assisted assessment), web based Learning Outcomes + assessment Learning Outcomes + assessment methods Learning outcomes + evaluation Bloom + assessment Bloom + Learning Outcomes + assessment Assessment or e-assessment or portfolios or multiple choice questions (MCQ) or essays or short answer questions (SAQ) and engineering, learning outcomes + engineering, learning outcomes + ABET, learning outcomes + EUR-ACE, learning outcomes + Bloom, ABET + Bloom, EUR-ACE + Bloom. The research was conducted using resources available at the library of School of Engineering, including books, journals and databases of journals and conference proceedings. Additionally, online searches were performed using web tools including Google, Google Scholar and specific websites. Table 2 presents the main sources used during this research project.. 10.

(33) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. Table 2 - Types of sources used and examples Types of sources Books. Journals. Conference proceedings. Policy papers. Websites. 2.2. Examples Taxonomy of educational objectives [41] A taxonomy for learning, teaching, and assessing: a revision of Bloom's taxonomy of educational objectives [40] Teaching for quality learning at university: what the student does [42] Assessing student learning in higher education [7] Assessment for learning in higher education [43] Designing better engineering education through assessment [44] Contemporary perspectives in e-learning research : themes, methods, and impact on practice [24] The Sage handbook of e-learning research [21] The e-assessment handbook [26] Assessment and Evaluation in Higher Education European Journal of Engineering Education Journal of Engineering Education British journal of educational technology Alt-J, research in Learning Technology Studies in Higher Education CAA Frontiers in Education Teaching and Learning in Higher Education EDUCOM Computers in Education European Commission Bologna Process Copenhagen Process National Institutions ABET EUR-ACE AHELO Tuning JISC E-learning Papers. ASSESSMENT. Assessment is an important process in education and intimately related to how students learn, as pointed by different authors. Brown, Bull and Pendlebury [7, p. 7] state that assessment is at the centre of the learning experience and should be a concern for those involved in the learning process, including learners, educators and institutions. In their view, assessment defines what students will consider important, how they will spend their time and even how they will see themselves in the future. They consider that to change learning we need to change the way we assess. This perspective is shared by Biggs [42] when he describes the backwash effect of assessment. Biggs defends that students will look strategically at assessment to determine what and how they will learn. Again, the main idea is that assessment is the central driver for the learning process. Another author, Peter Knight [43, p.11] shows a similar but more dramatic view when he asks how we can make student work without assessment. In the introduction to the book “Assessment for Learning in Higher Education” Knight considers that assessment is a moral activity that states the values of the teachers and institutions. What is assessed and the methods used give clear indication to the students of what is valued in a course or in a programme. In the view of the author, even though the goals of a. 11.

(34) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. full programme are stated in documents like the mission statement and the programme goals, it is in fact in the assessment tasks that lays the essence of the programme and of the learning experience. In the same book, David Boud [3] considers that assessment is the aspect of HE where there is more bad practice and the effects of this bad practice has a strong impact on students learning and success. As the author refers, students may avoid bad teaching methods but they cannot avoid being assessed if they want to graduate. Boud, based on the work of other authors considers that as a consequence of bad assessment practices, students may be hurt in their self-esteem and even reject some subjects. Race [45] also explores the negative effects of assessment, in particular exams, in terms of feelings. He describes the worst nightmares students get before or after an exam and how these bad feeling affect learning. Reinforcing this idea, Brown et al. [46] write that even though assessment is an important element of the learning process, new or existing faculty rarely have training to improve their assessment skills. An interesting perspective from the works of these authors is that in most cases, assessing and marking are activities done privately and most of the times teachers only receive feedback from students and not from their peers (other teachers) to validate their decisions. From the perspective of the students, the problem is similar. Most of the times students are assessed only by their teachers and don’t receive feed-back from other students. Race [45] goes even further and considers that one of the principles of assessment is the need for transparency not only to students and staff but also to the employers. In fact, recent calls for accountability of the institutions placed assessment on the main stage of the educational process. Assessment provides important evidence for quality and accreditation, which means that assessment methods, criteria and results won’t be as private as before. Even though some statements of these authors may be consider a bit strong, there are some interesting points made concerning the importance of assessment that may be summarized as follows: • • • • • •. Assessment is one of the main drivers for learning Has a strong impact on what and how students learn Affects students self-esteem and confidence Embeds the values of the teacher and the institution Is a compulsory activity to obtain a graduation/certification Provides evidence for quality and accreditation. Current trends, as Knight, Boud and other authors refer [3, 7, 43, 47] treat assessment as an activity that is part of the T/L process and that cannot be separated. As Erwin puts it [47] deciding what to teach and assess is one issue, not two. Assessment should be seen as a learning activity, centred on the learner. Brown et al. present an overview of current trends in assessment, focused on practical issues. A summary of these trends is represented in Table 3. It can be said that these changes are somewhat related with the overarching trend of student centred learning. These trends focus on promoting the formative function of assessment, increasing feedback to students and contributing to improve learning. It can be said that assessment is becoming more personalized and students are becoming more involved in the learning process through the use of explicit criteria and learning outcomes, allowing self and peer assessment and facilitating the recognition of prior-learning. From the trends referred in Table 3, three are of particular importance to the present work: the change from objectives to outcomes, from content to competences and from implicit to explicit criteria. As referred by Brown et al. using learning outcomes is useful to clarify the relationship between course. 12.

(35) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. design and assessment. Also, LOs play an important role in the recognition of prior learning since it separates learning from teaching activities, opening the possibility of assessing what was learned independently of how it was learned. Table 3 - Trends in assessment. From Brown et al. [7, p.13] From. Towards. Written examinations. Coursework. Tutor led-assessment. Student led-assessment. Competition. Collaboration. Product assessment. Process assessment. Objectives. Outcomes. Content. Competencies. Course assessment. Modular assessment. Advanced levels. Assessed prior learning. The change from content to competences is related with the change to outcomes. As referred by Brown et al. competences are clusters of skills students are able to use in different situations and they provide a framework for defining LOs and transferable skills. Finally the use of explicit criteria in assessment plays an important role in the pedagogical process. When the assessment criteria or marking schemes are explicit they may provide important information for the learning process. If shared with students, these instruments will give them indication of what is expected from them. By defining explicit criteria for an assessment task one is also defining the links between the task and the iLOs. Explicit criteria may have an important role in the reliability of the assessment task since it may reduce differences between assessors. As referred by Brown et al. [7] there is considerable controversy around the efficiency of the use of implicit and explicit instruments. However, the focus of this work is not on the instruments but on the methods of assessment. Boud [3] provides a useful description of the evolution of assessment that provides some background understanding on this matter. In the conventional conception, assessment follows learning and aims at finding out how much was learned; it is a quantitative perspective. There is no questioning concerning linking the assessment task to learning. This conception is followed by educational measurement that follows the same principles but it intends to be more rational, more efficient and reliable. It includes ideas and concepts from psychometrics. Nowadays we are still influenced by this conception as can be seen by the wide use of multiple-choice question exams that are a typical instrument of educational measurement. The latest perspective identified by Boud is competency based and authentic assessment. It resulted from concerns about validity of assessment focusing on the link between what was assessed and what was expected for students to have learned. Authentic assessment includes the direct assessment of complex performance and includes methods such as portfolios, open-ended problems, hands-on lab work. It contrasts with indirect assessment methods like multiple-choice questions that measure, among other things, indirect indicators of performance [48]. This conception of assessment questions the validity of educational measurement approaches and promotes performance-based assessment and the importance of learning outcomes. As Boud refers,. 13.

(36) Evaluation of the Application of e-Learning Methodologies to the Education of Engineering. what is important is to assess if the students are achieving the iLO, independently of how they reached them. Good assessment is, as described by Boud, the one that is linked with the iLO and the one that promotes learning. Erwin [47] also supports that the first step in educational design is to define clearly the learning outcomes. This should be done before deciding teaching and assessment strategies, both at programme and course level. Another author, Race [45, p. 67] provides ten principles of assessment and again, the first ones are related with clearly defining the purpose of assessment and integrating assessment in the course activities and not as a separate event. Race defends the importance of assessment providing feedback to students, in agreement with the perspective of Boud. Race and other authors [46] consider that a key question about assessment is knowing what we are assessing. However, the analysis proposed by these authors is not based on the iLO but on the assessment tasks. For each task it should be clear what is being assessed: the content, the process, the structure, the product, the style, the presentation, etc. The authors recommend that assessment criteria should be clearly stated and then provided and explained to students. They even go further saying that these criteria should be negotiated with the students, to share the ownership of assessment and help them understand the whole assessment process. Another interesting perspective is provided by Brown et al. [46, p.82] when they suggest that the outcome of an assessment task should not be merely a grade but a description of what students know and can do. Again, the link between assessment and LO is highlighted as being valuable in the educational process for the future development of students. Given the impact of assessment in student learning, Boud considers important to reflect on how assessment affects learning, what students learn from assessment. He considers assessment gives a message to students about what they should be learning but the message is not clear and most likely will not be interpreted the same way by teachers and students. For this reason, assessment will most likely have non-intended consequences in student learning. Students will respond strategically to assessment based on past experiences, choosing an approach that will lead them to success. Linn [48, p. 16] shares the same opinion saying that assessment may have intended and unintended effects both on learning and on teaching. As an example, both teachers and students may spend more time teaching or studying concepts that will be explicitly included on the assessment tasks and neglect those that will not be included. This concept is called consequential validity and is approached by Linn and other authors like Messick [49]. This is related with the backwash effect of assessment on learning. As Boud explains, the backwash effect is positive if encourages the intended learning outcomes and is negative when encourages ways of learning that are not desired, like memorizing instead of understanding. Linn [48, p. 19] suggests that to understand the real cognitive complexity of an assessment task one must analyse the task, the familiarity of the student with the task, and ideally the process students follow to solve it. An apparently complex task may be addressing lower level thinking skills if the student is only recalling previous knowledge about the task. To summarize, the following ideas are important when defining assessment: • • • • •. Assessment should be seen as part of the teaching and learning process (T/L) The first step is to have good assessment is to define the iLOs of the course or module It is important to define clearly the assessment tasks and what is being considered for assessment Assessors have to realize that assessment tasks will have intended and unintended outcomes The real cognitive complexity of an assessment task depends not only on the task but on many other factors, some related with the learner, others with the T/L process.. 14.