Evaluation of nursing practice and nursing education

(1)

EVALUATION OF NURSING PRACTICE

AND NURSING EDUCATION’

Patricia A. Deimanz

Evaluation is a vital part of nursing management. This article describes the major focuses of evaluation in nursing, principle steps in the evaluation process, and requirements for effective evaluation that in turn give rise to important implications for members of the nursing profession.

Introduction

Evaluation is a timely topic, one now receiving considerable attention from nursing groups around the world. Your selec- tion of “Methods of Evaluation” as the theme of the Eighth National Congress of the Nurses’ Association of Chile’ evidences concern about ways of determining the quality of both nursing and educational programs, and about ways of establishing quality control measures.

The nursing profession in Chile, as else- where, has discharged its responsibilities for ensuring quality to a certain degree by: (1) establishing requirements for those who enter the profession; (2) controlling the programs that prepare nursing personnel; and (3) confirming that those who complete these programs are competent -by means of examination and subsequent official recog- nition. However, no members of a profession who hold themselves accountable for the practice of that profession can be satis- fied by the mere establishment and enforce- ment of such entry-oriented controls.

‘Paper prepared for the Eighth National Congress of the Nurses’ Association of Chile held at Temuco, Chile, on 4-9 September, 1977. Also appearing in Spanish in the R&tin de la Oficina Sanitaria Pan- americana 85(6):487-497, 1978.

zPAHO/WHO Nursing Adviser at the time this paper was prepared. At present Nursing Adviser, Division of Nursing, BHM, HRA, U.S. Department of Health, Education, and Welfare.

Evaluation reinforces professional ac- countability. Nursing practitioners are accountable for the quality of their work- accountable to themselves, to their patients, to the patient’s families, to other health team members, to the institutions and agencies involved, to the community, to the nursing profession, and by way of the nursing profession to society as a whole (I). By the same token nurse educators are accountable to their profession, their teaching institutions, their students, and themselves. In both cases, however, it is only possible to be accountable, which is to say answerable for quality, if the quality of one’s work is actually determined through evaluation.

The Evaluation Concept

In essence, evaluation is the process of determining the degree to which preselected objectives have been attained. Evaluation is an integral part of the management process, which can be viewed as encompassing the four steps of planning, organization, imple- mentation, and evaluation. As this sequence suggests, evaluation not only presupposes but relies on initial planning. A meaning- I ful evaluation simply cannot be performed unless the planning needed to define objectives has taken place.

Evaluation also requires the use of value judgments. Such judgments, made through- out the evaluation process, help de-

(2)

“Sound observation.

. . is not

for the sake of piling up miscel-

laneous information

or curious

fact, but for the sake of saving

life and increasing

health and

comfort”

-F.

Nightingale

(3).

(Photo: WHO/P.

Almasy.1

e

fine what behaviors,

products,

or outcomes

are desired, how these things should be ob-

tained, critical

elements to measure, the best

instruments

with

which

to measure

them,

and what decisions should

be reached

as a

result of the evaluation

(2).

At

the same time, standards must be estab-

lished in order to determine

how well the

desired objectives have been attained.

Natu-

rally,

such standards

are a prerequisite

for

evaluation.

Overall,

it is important

to recognize

that

evaluation

cannot

be performed

by merely

describing

a situation.

Evaluation

requires

collection

of data,

interpretation

of the

data, and recommendation

of actions indi-

cated

by that interpretation.

In addition,

for evaluation

to be truly

productive,

em-

phasis must be placed

on those indicated

actions

-actions

designed

to support,

strengthen,

or otherwise

modify

the services

involved.

Florence

Nightingale,

writing

about the

related process of observation,

expressed this

point very succinctly.

She said:

In dwelling

upon

the vital

importance

of

sound observation,

it must never be lost sight of

what observation

is for. It is not for the sake of

piling

up miscellaneous

information

or curious

facts, but for the sake of saving life and in-

creasing health and comfort (3).

What

this says of observation

applies

equally

well to evaluation-for

no evalua-

tion can be of any practical

worth

unless

good use is made of its results.

Different

Applloaches

to Nursing

Evaluation

(3)

ing the nursing care or teaching processes involved, and by assessing the end results or outcomes obtained. For convenience, the three are commonly referred to as “structure,” “process,” and “outcome” evaluations.

To be a little more specific, structure evaluations generally consider the purpose, organization, fiscal resources and management, personnel qualifications, and physical plant and equipment of the institution, agency, or program being assessed. Process evaluations (I) appraise the steps taken to care for patients or educate students for the purpose of attaining specified therapeutic or educational goals. And outcome evaluations assess the patient care or teaching process in terms of its end result-that is, in terms of the measurable impact the process has on patient health status or nursing student behavior.

Generally speaking, these three types of evaluation require different methodologies and help answer different questions. Rele- vant questions to which each type is ad- dressed are as follows:

l

Structure evaluation:

Are the facilities,

equipment, and manpower resources con- ducive to effective teaching or delivery of quality care?

l

Process

evaluation: Is the nursing pro-

perly practiced? Or is the manner of teaching appropriate and adequate?

l

Outcome evaluation:

What effect does

the nursing care or instruction have on patient health status or student performance?

As these questions suggest, structure evaluation places the focus on characteristics of practitioners and the setting; process evaluation focuses on practitioner performance; and outcome evaluation is focused on the effects of the process on the patients, clients, or students.

These distinctions make it relevant to ask how the different types of evaluation relate to one another. Specifically, are the variables appraised by one more significant than

the variables appraised by the others? And, in a similar vein, does it suffice to employ one type of evaluation alone, or should balanced use of the different kinds be made?

Because of the general nature of these queries and lack of extensive scientific data, the available answers are necessarily empirical. However, a few studies reported in nursing journals have pointed ?rp the close relationships existing between structure, process, and outcome evaluations. The relevant findings of three such studies are briefly summarized below:

l Hegyvary and Hausmann (4) found a

number of structural variables to positively influence the quality of various components of the nursing process. In essence, these structural elements were: (1) organizational structure of the nursing unit; (2) the style of nursing leadership; (3 ) unit staff attitudes; (4) supervisory staff attitudes; and (5) graduate nurse preparation.

l A randomized trial of nurse praction-

ers providing primary health care focused attention on two outcomes-clinical effectiveness and safety. These outcomes were assessed in terms of the physical, emotional, and social conditions of two groups of patients-one receiving primary care from nurse practitioners and the other receiving conventional care. Besides demonstrating comparability of these two groups at the start of the study, the results showed similar levels of physical, emotional, and social functioning by both groups after receiving care for one year. In other words, the study showed the nurse practitioners to be providing care that was both safe and effective (5). This outcome study thus provided an excellent basis for exploring the firocess of primary health care delivery by nurse practitioners.

l Yauger (6) has reported on an outcome

(4)

findings showed that the family-centered approach to health care had no significant impact on health outcome. These negative findings pointed up an urgent need to evaluate the process involved. Or, as the author put it, “It is essential that the process of family-centered care be more closely identified and evaluated. An intervention that is not producing results must either be changed or the whole concept eliminated”

(6).

As all these studies suggest, it is important for appraisals to be made on the basis of all three elements of evaluation-structure, process, and outcome-rather than being restricted to just one or another element. Combining the results of all three types of evaluation will give a more complete picture of the quality of the services.

The Evaluation Process

The model of evaluation shown in Fig- ure 1 depicts it as a cyclical process con- sisting of the following general steps: (1) determine

what

is to be evaluated; (2) define evaluation

criteria;

(3) plan the meth-

odology

to be applied; (4) gather informa-

tion; (5) analyze the

results;

(6) take action;

and (7) re-evaluate.

Determining What Is To Be Evaluated

As the foregoing indicates, it has proven useful to have nursing evaluations focus on structures, processes, and outcomes. Logi- cally, then, the first step in planning an evaluation is to determine which of these are to be evaluated in order to achieve the desired aims. This implies, of course, that the aims of the evaluation must be defined initially so as to ensure consistency between the reasons for the evaluation and what is being evaluated. For example, if the evaluation seeks to identify strengths and weak- nesses of personnel performance, the main focus of the evaluation must clearly be on processes, although evaluation of structures and outcomes can be included as well. Once determined, this primary focus of the evaluation will play a large part in dictat- ing who should be responsible for carrying out the subsequent steps of the evaluation process.

Defining Evaluation Criteria

Generally speaking, a criterion used for evaluation purposes should constitute a baseline for measuring some event or behavior (7) that is “established by authority, custom, or general consent” (8). Such a criterion should also be a single element in a given situation-an element that is clearly

Figure 1. A model of the evaluation process.

(5)

defined, measurable, and relevant to the situation or condition being assessed. These latter qualities are essential to the performance of an objective evaluation that will yield valid results.

Structural criteria. Criteria must necessarily be developed in accordance with the focus of the evaluation. Structural criteria, with which we are particularly familiar, are elements relating to: the purpose of an institution, agency, or program; organizational characteristics; fiscal resources and management: personnel qualifications; and physical facilities and equipment. Two examples of structural criteria are that a graduate nurse supervisor is present on each tour of duty at a hospital, or that an educational institution has at least one instructor for every 10 nursing students.

Process

criteria. These criteria, which relate to the performance of nursing personnel or nurse educators, help measure such factors as: the nature of interactions between the nurse and the patient or client, or the nurse educator and student; the extent to which objectives are realized; the degree of skill with which techniques or procedures are executed; and the degree of consumer participation. Examples of process criteria are that the prenatal patient receives at least 10 hours of instruction; that the nurse develops a care plan for each patient; and that the instructor provides for course evaluation on the part of the students. Ideally, logically developed process criteria would permit a qualitative judgment concerning proper completion of each phase of the nursing care process. Using such criteria, the evaluator could determine the accuracy with which the patient’s needs are identified, the appropri- ateness of the goals set, the adequacy of the interventions planned, and the skill with which these interventions are performed (7).

Brief mention should also be made of performance appraisals. Unfortunately,

most formats for evaluating personnel call for judgment of personal characteristics such as appearance, initiative, quantity of work, and competence -rather than of actual performance. Nevertheless, the merits of measuring individual performance against criteria established in accordance with job expectations can hardly be questioned. And although formulating criteria to judge the quality of role performance is a time-consuming task, nurses who have done it report that the positive results obtained more than compensate for the time and energy required.

Outcome criteria. The third major type of criteria, outcome criteria, focus on measurable alterations in patient health status or student behavior resulting from health care or the educational process.

Health outcome criteria are unquestion- ably the most difficult to draft, due princi- pally to the multiple causes of health effects. It may be difficult, for example, to determine if a case of wound sepsis was the fault of the surgeon, the nurse who changed the dressings, or personnel in the central sterile supply room. By the same token, if a surgical patient remains free of infection, it is not due to the efforts of nursing alone.

The problem of multiple causation, however, is surmountable. Stevens suggests developing outcome criteria applicable to groups of patients for whom similar outcomes are desired. That is, criteria could be formulated for patients with the same disease (e.g., cardiac patients), for those receiving similar treatment (e.g., preopera- tive patients), for those with like needs (e. g., geriatric patients), or for those at the same stage of an illness (e.g., patients re- quiring intensive care) (7).

(6)

a performance criterion is that the patient can identify his or her medications, while a state of being criterion is that the patient has intact skin in the affected area. In this regard, it is noteworthy that both types of outcome criteria could be formulated for any of the groups of patients mentioned in the preceding paragraph.

Develofiing and

checking

criteria. A relevant question at this point is who bears the responsibility for drafting criteria? Does it rest with nurse educators, nursing service personnel, or nursing administrators? My own belief is that development of structure, process, and outcome criteria should be done by all three groups working together. Nurse administrators should join forces with nurses with expertise in caring for pediatric patients and with instructors of pediatric nursing to develop pediatric criteria; and the same approach should be followed in other specialties. In fact, this same collaboration should occur regardless of whether the criteria developed are to be applied in the practice or educational settings. The underlying reason for this is that criteria should be developed by a rep- resentative group of nurses who have expertise in administration, provision of care, and education in the particular field concerned.

In addition, I am aware of one instance where auxiliary personnel were also involved in defining criteria. This had the gratifying effect of improving staff relationships and unifying the staff in their efforts to do something about the quality of the care being provided.

Another significant point is that once criteria have been developed, the validity of those criteria needs to be checked. One way of doing this is by peer review, a method that yields added benefits by in- volving more nurses in the evaluation process, arousing their interest and enthu- siasm, and thereby enhancing implementa- tion of quality control efforts.

Planning the Methodology

A format in keeping with the purpose of the evaluation must be prepared for gather - ing the information desired. In addition, decisions must be made about the source of the information, how and when the information will be collected, and who will collect it. These decisions, especially the two former ones, should reflect full consideration of the need for accurate and efficient data collection.

Data sources may be direct or indirect. The former include direct observation of nursing performance and the results of nursing activities while reports by third parties, records, and reports by those who are the subject of evaluation fall into the latter category. In general, indirect sources tend to be less reliable, have more biases, and provide less valid information than direct sources (9).

Despite the fact that the patient, client, or student is an indirect data source, the importance and value of consumer participation in evaluating the services received needs to be emphasized. For although the consumer may not have the background to assess care or instruction professionally, he or she is the

only

person who can judge whether his or her perceived needs were met.

Various methodologies and data sources appropriate for structure, process, and outcome evaluations are shown in Table 1.

Gathering Information

The people responsible for collecting information should receive adequate orienta- tion-irrespective of how simple the methodology employed may seem.

(7)

Table 1. Selected methodologies and data sources employ4 in structure, process, and outcome evaluations.

Focus of

evaluation What to assess

Appropriate

methodologies Data sources

Structure Purpose and objectives Organizational character-

istics

Manpower resources Fiscal resources and man-

agemen t

Physical plant and equipment

Process The performance of the practitioner or educator: i.e., the ways care or instruction is provided

outcome End results or effects of the care or education received

Observation Use of check lists Use of questionnaires Interviews

Review of records, reports, and documents

Task analysis Nursing audit Observation

Use of video and audio tapes Demonstrations

Oral and written commu- nications

Use of questionnaires Observation Nursing audit Interviews Demonstrations

Oral and written commu- nications

Nursing personnel (practi- titioners and educators) Physical plant

Nursing unit

Records, reports, and documents

Patient/client records Patients/clients Nursing students

Nursing personnel (practitioners and educators)

Patient/client records Patients/clients Nursing students

course and at its termination. Evaluation of student performance may be carried out whenever the student is in the clinical area-by means of self-assessments and instructor assessments employing anecdotal notes. Performance appraisals of staff members are ordinarily done more frequently during each member’s probation period and less frequently thereafter. Three occasions that might be considered appropriate for assessing outcomes of hospital patient care are: (1) at the end of the critical stage of the patient’s illness; (2) upon the patient’s discharge from the inpatient unit; and (3) one or two months following discharge (10).

Analyzing the Results

Both interpretation of the data and feed- back to all individuals concerned should take place within the shortest time feasible

once the information has been gathered. In addition, opportunities should be provided for discussing the evaluation’s results-particularly if those doing the reviewing and those reviewed are different parties.

Taking Action

Evaluation results may indicate a need to reinforce what is already being done, to modify procedures, or to introduce other changes. They may also call for shifting priorities, revising objectives, or developing new programs or services to meet previously unidentified needs.

(8)

referred to appropriate authorities; and here it must be recognized that the likeli- hood of receiving administrative approval and support is far greater when the requests for change are well-documented by the results of a sound evaluation. To cite an obvi- ous case, a request that additional funds be spent on supplies might not succeed if the requesting parties give the impression that they wanted a well-stocked supply closet. Success would be more likely if the request were accompanied by a solid evaluation report which includes the incidence of post- operative wound infections.

Re-evaluation

When actions designed to improve the situation have been completed, another evaluation must be made in order to determine whether or not they have had the desired effect. To state this point more generally, evaluation is a cyclical process requir- ing continual repetition. And it is only to the extent that the nursing profession recog- nizes this fact and acts accordingly that continuing improvement in the quality of nursing programs can be assured.

Implications for Nursing

What has just been said about evaluation in general has some very specific implications for those of us in nursing today. I would like to briefly mention four that appear most important.

l

The need to define @rogram or service

objectives in terms

of

health outcome.

Just as defining the behavioral outcome objectives of an educational program make the program more amenable to assessment, so the definition of health outcome objectives would facilitate health program evaluation. Such defined objectives, describing the desired health status of the patient or client served, in either intramural or extramural settings, would better enable nursing per-

sonnel to gauge the effectiveness of the services provided. All nurses, but particularly those in the service setting, need to give this point serious consideration.

l

The need

for

systematic

evaluation of

role changes.

The changing needs and de- mands for health services exert a profound influence on the nurse’s role-a role that over the last decade has tended to expand. This makes it essential that changes in the roles performed by nurses be planned for and systematically evaluated. Not only is this necessary in order to provide well- articulated nursing services and appropriate nursing education, but it is also necessary in order to determine whether the nursing needs of society are actually being met.

l

The

need

to define relevant national

criteria.

It is imperative for nurses to define criteria that will be relevant for judging the quality of nursing practice and education - within the all-important context of their respective country’s social, cultural, and economic framework and health care delivery system. This task might not be as Hercu- lean as it sounds if special groups of nurses were formed and each one assigned responsibility for defining criteria in a specific area. This process could be assisted by having the national nurses’ association develop a coordinating mechanism for sharing criteria developed by nurses from a variety of agencies and educational institutions.

l

The need

for

nursing leadershi@

of

(9)

SUMMARY Evaluation is a basic part of nursing management. Though there are many types of evaluation, it is possible to discern three predominant focuses in nursing, each viewing nursing activities in a different way. These are “structure” evaluations which are concerned with the administrative and other structural elements underlying nursing activities; “process” evaluations which are concerned with nursing activities themselves, and “outcome” evaluations which are concerned with the end results.

While it is difficult to generalize about the relative merits of each kind, it seems clear that wherever possible all three types should be employed. That is because such combined evaluation is capable of giving the truest picture of the quality of the services being provided.

Another basic point is that evaluation is a cyclical process that must be repeated. The basic steps involved are as follows: determine what is to be evaluated, define evaluation criteria, plan the methodology to be applied, collect data, analyze the results, take action, and re-evaluate the effects of that action.

Because ongoing evaluation is necessary for continuing improvement of the quality of nurs-

ing service and educational programs, the nursing profession must be cognizant of and take appropriate action with respect to the following needs:

l The need to define program or service ob-

jectives in terms of health outcome. Such defined objectives would better enable nursing personnel to gauge the effectiveness of the program or services involved.

l The need for systematic evaluation of

changes in the nurse’s role. This is necessary in order to provide well-articulated nursing services, appropriate nursing education, and assur- ance that the nursing needs of society are actually being met.

l The need to define relevant national crite-

ria. This is imperative for judging the quality of nursing care and education within the context of each country’s social, cultural, and economic framework and health care delivery system.

l The need for nursing leadership. Where

there is no multidisciplinary quality control program, nursing administrators must frequently assume a leadership role and establish quality control programs that will monitor the care being provided under their direction.

REFERENCES

(1) Phaneuf, M. C. The Nursing Audit: Pro- file for Excellence, Appleton-Century-Crofts,

New York, 1972, pp. 6 and 15.

(2) Arndt, C., and L.M.D. Huckabay. Nurs- ing Administration. C.V. Mosby, St. Louis, 1975, p. 136.

(3) Nightingale, F. Notes on Nursing. J. B. Lippincott, Philadelphia, 1946, p. 70.

(4) Hegyvary, S. T., and R. K. Hausmann. Correlates of the quality of nursing care.J Nurs Admin 6(9):22-27, 1976.

(5) Sackett, D. L., W. 0. Spitzer, M. Gent, et al. The Burlington randomized trial of the nurse practitioner: Health outcomes of patients. Ann Intern Med 80(2):137-142, 1974.

(6) Yauger, R. A. Program evaluation in com-

munity health nursing: Does family-centered care make a difference? Nurs Outlook 20(5):320- 323, 1972.

(7) Stevens, B. The Nurse as Executive. Con- temporary Publishing, Wakefield, Mass., 1975, pp, 128, 149, and 151.

(8) Ramey, I. G. Setting standards and evaluating care. J Nurs Admin 3(3):27-35, 1973.

(9) Hagen, E. Conceptual issues in the ap- praisal of the quality of care. In: U.S. Depart- ment of Health, Education, and Welfare, Assess- ment of Nursing Services, Bethesda, Maryland, 1975, p. 61.