Final M&E curriculum document July 2007

(1)

Introduction

The process of implementing successful health programs has many steps: clear identification of the problem, identification of feasible solutions, priority setting, determination of the types and quantity of inputs, determination of the type and quantity of human resources, establishment of monitoring systems, formative research to design and adapt materials and resources, and other steps. At each step in this process and to improve the quality of services, these activities need to be managed through a well established monitoring and evaluation system. Monitoring and Evaluation in the delivery of health care is essentially a separate discipline in public health that involves management, social sciences and epidemiology. It involves a systematic assessment of health programs and their progress towards goals. Specific tools and approaches, strategies, and the experience of more than forty years of specific applications define this new field. Separately seen, monitoring refers to the routine or the day–to-day collection of data about the health program’s ongoing activities or specific behaviors in a population. Evaluation may build on monitoring, but it is an independent activity, focusing on addressing specific questions about the performance of a program, an HMIS or the adequacy of planning. Evaluation differs from monitoring because it arises in response to specific programmatic needs. Shaping a unique response in every instant requires exceptional scientific and programmatic skills. Although monitoring and evaluation (M&E) became a major new field of scientific endeavor primarily because of issues of accountability used to monitor the expenditure of public or private funds, it has grown into a major scientific movement incorporating management and organization science, economics and other social sciences and epidemiology. It has been demonstrated that

(2)

M&E plays a critical role in the regular improvement of the quality of services and programs.

Monitoring and evaluation of a health program or a national health delivery system refers has been defined as the application of scientific methods intended to systematically assess performance in terms of organization, cost, process, output, outcome and impact. Higginson (1994) defines several broad forms of M&E, including organizational M&E, intending to assess the organization and flow of health care delivery; clinical and medical M&E focus on the measurement of output and outcome of care. Two other broad classes of M&E can be noted: the M&E of policy and policy decision-making, and the M&E of the impact of programs on specific diseases and overall levels of health. It is now considered an essential tool for all interventions.

For M&E to be an effective management tool, therefore, there should be three important and essential elements: An M&E system that clearly identified and embraced a set of highly relevant, efficient and coherent performance indicators. The availability and design of such M&E system is critical in the measurement of the program impact both during the execution and the post-evaluation phases. The availability of such system will allow the ascertainment or verification of whether the objectives identified in the health program design phase were or will be effectively achieved. The second important element is the availability of a well organized and effective HMIS system. This implies the availability and hence the implementation of a health information system throughout the health delivery system that ensures the timely availability of quality data and other relevant documentation based on the needs. Without such a system, it will be very difficult to ensure a structured M&E system that could be robust and reliable. The third most important element is the M&E structure the inevitable availability of a highly trained corps of health monitoring and evaluation

(3)

professionals to help design and operate this system, and to interpret and use the data to help decision-makers improve policies and programs. These are critical issues and the one appearing to be the greatest challenge for many health care systems in developing countries.

Although M&E has often been presented as an application in specialized professions such as epidemiology, health economics and in other social and clinical areas, it is only since the 1980’s that M&E has been established as a stand-alone profession, with its own methods and theories. In view of the pressing and the growing pressure on the health sector of Ethiopia to closely monitor and evaluate government interventions in relation to poverty reduction including health MDGs, the absence of appropriate professionals in the area of health M&E should undoubtedly be seen as the main trigger for initiating such professional training in Ethiopia.

Cognizant of the growing needs for a professional M&E specialization in the health care delivery system of Ethiopia, Jimma University in collaboration with its national and international partners has become the pioneer amongst the Universities in Ethiopia in starting a post– graduate program leading to both a Post-graduate Diploma and an MSc course with the aim of fulfilling the long-term requirement of the National Health Care System of Ethiopia in the area of M&E specialization. This course is the first of its kind in Africa, and only second in the world after the initial start and success in Brazil under the leadership of the Brazilian National School of Public Health – Oswaldo Cruz Foundation ( FIOCRUZ).

The design of the training program will incorporate skills and content from various areas of health science, including epidemiology,

(4)

biostatistics, management, and health economics. It will focus on four areas (called Themes, below) we have defined as: The Social, Cultural, Political and Economic Context of Health; Logical Models of Health Programs; Logical Models of Evaluation; and Communication, Networking and Information Technology. Students will also be rigorously trained in the application of new technology, including computer assisted learning, distance learning, use of the internet and website design, as well as traditional skills in writing and public presentation. The rigorous curriculum contains theoretical, community-based, and practical training.

Rationale of the Post Graduate Training program

Ethiopia is now committed to achieving the MDGs by the year 2015. Such commitments , especially in the health sector requires the close monitoring and evaluation of priority health programs, especially HIV/AIDS, Malaria, Tuberculosis, and maternal and child health . However many studies and reviews on the health sector in Ethiopia have shown that the weakest link in the implementation of HSDP is the lack of information about how programs are implemented and working. A well organized system of health data collection and use is called a Health Management Information System (HMIS), and is an essential component of an M&E system. In addition to these routine reporting systems, programs need evaluation to identify obstacles and develop solutions for improvement. It is not only that the HMIS and other elements of the Monitoring and Evaluation system are not well implemented nationally, but that there are virtually no professional Monitoring and Evaluation experts available to the health sector in the country. It is therefore very important for the health sector to acquire such expertise to provide critical leadership in the organization of an effective monitoring and evaluation system.

(5)

Objectives of the training

General Objective of the Post graduate Program

The general objective of the graduate program is to develop a corps of new professionals in Health Monitoring and Evaluation.

Specific Objectives of the training Program

To train M&E specialists who will provide critical leadership to achieve sustained improvement in the coverage and quality of health programs,

To develop M&E specialists who will be capable of understanding the organization and delivery of health programs, including the justification and assumptions embedded in such programs.

To develop M&E specialists capable of designing and implementing monitoring and evaluation plans for priority health programs, including health policy,

To develop M&E specialists, who at every level of the health system can advise in the use of data for evidence based decision making

To develop M&E specialists capable of using new technologies to achieve program goals.

To train a cadre of M&E specialists who will professionalize the field of M&E in Ethiopia, Africa and elsewhere and lead the successful development of this field.

Philosophy, Faculty and Teaching

Philosophy: the program adopts the educational philosophy of Jimma University - community based education - with the adult active learning strategies utilized by the National School of Public Health in

(6)

Brazil. The teaching staff will combine faculty from Jimma University with national and international specialists.

Teaching-learning methods

The teaching methodology uses innovative and adult teaching approach with active participatory learning and integrated training over the course of study.

National and international experts in health programs and M&E will present to the students to ensure adequate exposure and familiarity with global M&E systems.

The program will make available a computer lab, library services, access to the internet, and reference materials, including computer based and distance learning materials.

Graduate Profile

After completion of their training graduates will be:  Creative and well versed M&E professionals

 Capable of understanding programs that are relevant to and that address issues related to national public health priorities  Capable of understanding basic principles of monitoring and

evaluation including design, monitoring and evaluation tools, implementation of M&E systems, and the collection, analysis and critical appraisal of information

 Capable of understanding processes in program or project management

 Capable of developing an integrated and comprehensive Monitoring and Evaluation Plan

 Excellent at interpersonal skills, capable of organizing and influencing groups to use M&E data

(7)

Diploma/Degree Awarded

After successful completion of the taught course component and the evaluation proposal (normally during the first year) the graduates will be awarded the Jimma University Post graduate Diploma in Health Monitoring and Evaluation (JUDipME). The graduates of the 2nd_{year of}

the program, which includes the submission and acceptance of a written thesis, and successful completion of DTTP will be awarded a Master of Science (MSc.) in Health Monitoring and Evaluation.

Students must maintain a cumulative grade point average equal to or greater than 3.0 to continue in the course, as described in the graduate legislation. Students, even if maintaining this grade point average, but who fail the evaluation proposal, will not be allowed to continue in the course. They may be awarded a certificate of attendance.

Training to be offered

Training is organized around four themes: the socio-political-cultural context of health; logical models of health problems; logical models of evaluation; communication, networking and information technology.

(8)

1. Themes required for M&E of Health Programs

Theme Course Cr. hrs

Theme 1:

Social-Political-Economic and Cultural context of Health PUBH 641 Principles of Health Management 2 MEHP 602 Health Management Information Systems 1 Total 3

Theme 2: Logical models of Health Programs PUBH 611 Epidemiology I 2 PUBH 612 Biostatistics I 2 PUBH 614 Biostatistics II 3 MEHP 601 Priority Health Programs 1 Total 11

Theme 3: Logical Models of Evaluation MEHP 603 M&E 1 Introduction to Monitoring and Evaluation 1 MEHP 604 Evaluability Assessment 2 MEHP 605

M&E 2 Methods and Models for Process Evaluation

3

MEHP 606

Health Economics and Economic Evaluation

(9)

MEHP 607

M&E 3 Methods and Models of Outcome and Impact Evaluation

3 MEHP 608 Evaluation Research Ethics 1 MEHP 609 Meta-evaluation 1 MEHP 615 Evaluation Proposal 3 Total 15 Theme 4: Communication, Networking & IT COMP 601 Computers and Applications 2 MEHP 611 Advanced Software Applications for Health and Project Management 1 MEHP 612 Proposal Development 1 MEHP 614 Writing an Evaluation Proposal 2 Total 6 DTTP 6 Thesis 6 Total 44 2. Theme Descriptions

Theme 1: Social-Political-Economic and Cultural context of Health

(10)

Describes the context of health and health systems in Ethiopia. It formally articulates the relations among the social, cultural, political and economic environments to health status, health behaviors, and the performance of the health system. These course provide an introduction to understanding how these distal determinants are connected to more proximate causes of disease, and how they set the foundation for the performance of the formal and informal health system.

Theme 2: Logical Model of Health Programs

At the core of the evaluator’s skill is the ability to understand the nature of health problems and the elements and sequences of activities embedded in the response to the health problem. This theme focuses on developing the knowledge and skills required to understand the nature and magnitude of priority health problems in Ethiopia, and to critically analyse the assumptions and activities of interventions. Necessary to understanding the nature and magnitude of health problems are the traditional core skills of public health: epidemiology and biostatistics, and they are included in this theme. The theme is called logical model of health programs because all the elements of the learning units contribute to the student’s ability to critically analyze health interventions and develop a logical model for the intervention or service. This logical model serves as the basis for designing evaluations of the program.

Theme 3: Logical model of Evaluation

This theme is the core of the Monitoring and Evaluation focus of the Diploma and Masters program. Students will be required to develop a special competence in evaluation theory and methods. These include skill sets taught in epidemiology, biostatistics, health informatics, economics and other sciences, but are adapted to the particular uses

(11)

and functions of M&E. At the completion of the course the students will be able to design monitoring systems and evaluations for priority health programs, collect and analyse data and develop recommendations for programs.

Theme 4: Communication, Networking and IT

This theme incorporates a combination of skills from different areas that may be used to organize, transmit and utilize evaluation results. Evaluators must be equipped to work with diverse professionals to conduct evaluability assessments, stakeholder analyses and to conduct M&E activities. Evaluators must perform as neutral and supportive professionals in interpersonal work settings that may be conflictive. They must have skills to negotiate in these settings and to resolve personal and operational crises. Evaluators must also assist policy makers and program heads and staffs to accept and use the findings from M&E activities. Personal and professional networking skills are important for these tasks, but we will use the topic of interpersonal networking to build a national and international M&E network to support the activities of our students. Evaluators also need to know how to present findings concisely and effectively to a range of audiences. Finally, students will be exposed to the latest microcomputer and web-based programs, to be utilized in the planning, implementation, analysis and use of M&E data for program improvement. Literacy in these technologies will maximize the utility of these M&E professionals and equip them to respond to the multiple demands placed on them at their worksite.

(12)

The students will be evaluated using the following methods of academic performance appraisal:

Post Graduate Diploma in M&E

 Assignments as required including community based assignments

 Class attendance and active participation in the teaching-learning process

 M&E field work assignment and presentation of field activity report

 Seminar presentations

 Written examination at the end of each course  Preparation of a Evaluation Proposal

MSC in Health M&E (In addition to successful completion of taught course component and acceptance of evaluation proposal)

 DTTP

 Submission and acceptance of a thesis

Monitoring and Evaluation of the training program  Feed back from graduates

 Feed back from stakeholders

 Supervision and feedback from student mentors  Student scores on assignments and exams

 Quality of proposal, thesis and other written products and presentations

(13)

Developmental Team Training Program (DTTP)

Community Based Education activities need to be targeted to bring about development impact. Multisectoral teams of students from Jimma University will work together with the community and other development workers to develop interventions of use to the

community. Graduate students in Monitoring and Evaluation in Health will be assigned for DTTP to a community to identify health problems and design an implementation plan.

Student Research Project (Thesis)

Upon completion of the taught course component postgraduate students of Jimma University are expected to undertake independent research and present it in the form of a thesis. The thesis becomes an opportunity to actualize skills learned in previous training and demonstrate their ability to work more independently. Theses in this program involve participatory formulation of an evaluation question, designing an evaluation, developing research instruments, collecting data, analyzing data and presenting data. Theses will also involve a review of appropriate literature and substantial familiarity with an intervention and approaches to evaluating it. The research undertaken is the requirement for graduation with MSc Degree in monitoring and Evaluation.

(14)

Theme 1 Social-Political-Economic and Cultural context of Health

Learning Objective

At the end of the Unit, students will be able to:

 Understand the social, political, cultural, economic and environmental determinants of health and healthcare  Be able to work with different disciplines and stakeholders

including collaborators outside of the health sector Contents

 Comprehensive vs. Disease focused understanding of health  Non-proximate determinants of health: interlinkages among

health, poverty and social and economic development

BLOCK1-Unit 2: Ethiopian healthcare delivery and financing system

Learning Objective

Describe the organization of the Ethiopian healthcare delivery system

and the financing system and justifications Contents

History of healthcare delivery system in Ethiopia Organization of the healthcare system

Sources of health financing in Ethiopia, expenditures

(15)

Learning Objective

Understand the global and national health sector reform

movements within the context of structural economic adjustment policy

Contents

Health sector reform (HSR) in the context of economic reform in Ethiopia.

Major elements of Health Sector Reform

The private/public partnership in the delivery of health care The implication of Health Sector Reform on equity, access and

delivery of care

BLOCK1-Unit 4: Introduction to Health Services Management Learning Objective

Understand principles, concepts, and techniques of management in planning, implementation, and evaluation of healthcare

interventions

Be able to conduct a health profile of a given population or area Contents

Concepts and principles of management

Functions of management: Planning, Implementation and

performance evaluation and management of critical resources Managing change

Application of Business process Re-engineering (BPR) Leadership and decision-making

Negotiation and Conflict resolution Team building

(16)

BLOCK2-Unit 5: Policymaking and analysis in the Health Sector Learning Objective

Understand the process of national health policymaking and analysis

Understand the historical development, structure, and interaction of public health, healthcare systems, and national health policy Understand the process of public health policymaking and analysis Content

Operational definition of policy

The process and determinants of health policymaking Key operational levels of policy(Systemic, Organizational,

Programmatic Instrumental)

The Role of power in policymaking process

Application of evaluation criteria to health policy options BLOCK2-Unit 6: Strategic Planning in the Health Sector Learning Objective

Understand the definition and uses of strategic planning Content

Theory and types of health planning

Definitions of and approaches to strategic planning Steps in the development of strategic planning Setting organizational performance standards

BLOCK2-Unit 7: Introduction to Health Economics Learning Objective

Understand and apply the basic techniques and methods of economics to the field of health.

Content

(17)

The health care market: demand and supply of health care commodities.

Characteristics of health commodities, including agency relationships.

BLOCK3-Unit 8: Public Health Legislation and Ethics Learning Objective

Understand the public health legislation and code of ethics Content

Public health laws of Ethiopia Code of ethics for public health

(18)

Theme 2. Logical Model of Health Programs Credit hours = 11

Summary of Training Units

Theme 2: Logical models of Health Programs PUBH 611 Epidemiology I 2 PUBH 612 Biostatistics I 2 PUBH 614 Biostatistics II 3 MEHP 601 Priority Health Programs 1 Total 11

MEHP 601 Priority Health Programs Credit Hours: 1

Instructor: FMoH and partners. Responsible: M&E Unit, Department of Health Planning and Health Service Monitoring

Requirements: None Learning objectives:

Understand the national and global prevention and control programs for HIV/AIDS, Malaria, TB, MCH and other priority health programs

Course Content:

National/global health policies and programs Description of Teaching Strategy:

Presentations by experts, discussion of student experience with programs

Student’s Evaluation: Exam

(19)

Reading materials and articles provided by experts HSDP I-III

National Strategies

Theme 4. Communication, Networking and IT Credit Hours = 6

Summary of Training Units

Theme 4: Communication, Networking & IT COMP 601 Computers and Applications 2 MEHP 611 Advanced Software Applications for Health and Project Management 1 MEHP 612 Proposal Development 1 MEHP 614 Writing an Evaluation Proposal 2 Total 6

COMP 601 Computers and Applications Credit Hours: 1

Instructor: FMoH and partners. Responsible: M&E Unit, Department of Health Planning and Health Service Monitoring

(20)

Understand the national and global prevention and control programs for HIV/AIDS, Malaria, TB, MCH and other priority health programs

Course Content:

National/global health policies and programs Description of Teaching Strategy:

Presentations by experts, discussion of student experience with programs

Student’s Evaluation: Exam

Bibliography:

Reading materials and articles provided by experts HSDP I-III

National Strategies MEHP 611

Advanced Software Applications for Health and Project Management

Credit Hours: 1

Instructor: TBD Responsible: M&E Unit, Department of Health Planning and Health Service Monitoring

Requirements: COMP 601 Learning objectives:

Be familiar with appropriate software such as Epi-Info, Health Mapper, MS-Project, ArcView, etc.

Course Content:

Review of software. Software may change; currently Epi-Info, Health Mapper and MS-Project are used.

(21)

Introduction to the software by experts, hands on demonstration with the software. The course will normally be offered over a two week period.

Student Evaluation:

Successful completion of exercises and Exam Bibliography:

Software training guides MEHP 612

Proposal Development I and II Credit Hours: 1

Proposal Development I-Be familiar with library resources at Jimma, and internet search engines and sources in the health sciences. Proposal Development II-Be able to present their ideas in public to diverse audiences using good oral communication skills,

presentation software, video and other communication media. Course Content:

Expert presentation on communication skills and using presentation software.

Description of Teaching Strategy:

Presentation by experts and student presentations and expert critiques. The course will normally be offered over two separate time periods, with Proposal Development I offered as soon as possible when students arrive.

Student Evaluation:

(22)

Bibliography:

Public Speaking MEHP 614

Writing an Evaluation Proposal Credit Hours: 2

Correctly write a scientific document in English in word processing software, with correct formatting, footnoting and citation.

Course Content:

Presentation by expert and individual critique of writing assignments.

Description of Teaching Strategy:

Presentation by experts and individualized feedback and exercises in English.

Student Evaluation:

Assignments submitted by students Bibliography:

Writing a Proposal

(23)

Unit two: Writing and Presentation skills Unit three: Introduction to IT

Unit four: statistical Packages Block III

Unit one: Information Technology Unit two: Peer Networking

BLOCK1-Unit 1: Introduction to Health Education and Promotion

Learning Objective

Understand principles and approaches to health education and promotion and know how IEC/BCC interventions are

implemented Content

Key concepts (HE, HP, IEC, BCC, etc.)

Principles, components, strategies and practices of HE/HP

Theories in health education/health communication: a focus on BCC Planning, implementation and evaluation of HE/HP: a focus on

(24)

BLOCK1-Unit 2: Writing and Presentation skills Learning Objective

 Write a brief paper in English using scientific writing style Content

 Writing skills

 Scientific writing examples

BLOCK1-Unit 3: Introduction to IT and Ethiopian IT policy Learning Objective

At the end of the Unit, students will be able to:  Know IT policies of Ethiopia

 Identify different components of computers, and know how to use and care for a laptop

 Have a clear understanding of common hardware and software, and their interaction

 Use word processing, spreadsheet, and presentation programs  Be familiar with the internet and common search engines Content

 National telecommunication, data use, research ethics and IT policy

 Microsoft XP, MSOffice

BLOCK1-Unit 4: Introduction to Statistical Packages Learning Objective

At the end of the Unit, students will be able to:  Utilize soft ware for data analysis

Content

(25)

BLOCK2-Unit 1: Data for evidence-based decision making Learning Objective

 Understand how data can be collected and used to assist decision-making in different audiences

 Understand what can constitute evidence

 Be able to present data to diverse audiences clearly and concisely

Content

Case studies: data for decision-making

Dissemination of findings using different presentation techniques and approaches

BLOCK2-Unit 2: Writing and Presentation skills (continued) Learning objective:

 Undertake effective and accurate oral and visual presentation of demographic, statistical, programmatic and scientific information Content

 Guidelines for scientific presentation  MS Powerpoint

 Examples of effective (oral, written, etc) presentation and communication of information for professionals and lay audiences

BLOCK2-Unit 3: Introduction to IT (2) Learning Objective

At the end of the Unit, students will be able to:  Use MS Project and MS Access

(26)

 Microsoft Project  Microsoft Access

BLOCK2-Unit 4: Statistical Packages (2) Learning Objective:

At the end of the Unit, students will be able to:  Understand and use statistical packages Content  SPSS  EPIINFO  STATA  Health Mapper BLOCK3-Unit 1: Introduction to IT (3) Learning Objective

At the end of the Unit, students will be able to:  Design and build a simple website

 Be familiar with GIS software and GIS PDA applications Content

 MS Front Page and DreamWeaver  GIS

 PDA (service availability mapping –SAM)

BLOCK3-Unit 2: Peer Networking and Career Development in M&E

Learning objective:

(27)

 Describe the national and international community of M&E organizations and specialists

 Take advantage of regular professional interpersonal lead and participate in groups to address specific issues in monitoring and Evaluation

Content

 Key concepts (leadership, participatory learning and action, group dynamics, etc)

 Organizational mapping of M&E

 Establishing partnerships among GO/NGO, Private sectors and others

Block : 3- Unit: 3 : Introduction to Proposal design: BL 3 Learning Objectives:

Understand and apply the principles of proposal writing Contents

 Components of proposal writing