ELEMENTS OF A PROGRAM EVALUATION
What Is Program Evaluation?
By definition, program evaluation is "the systematic
application of social research procedures for assessing
the conceptualization, design, implementation, and utility
of social intervention programs." In practice, program
evaluation plays a significant role in program development
and assessment. From concept to planning, application to
results, the systematic evaluation of each step of a
program will serve as a mechanism to developing a
realistic program that is clear, comprehensive and
measurable. A solid program evaluation will also expedite
the dissemination and publication process. Some key areas
where incorporating program evaluation methods could be
- new program designs and development;
- program management and tracking;
- efficiency of program implementation;
- dissemination; and
- program effectiveness.
Developing An Evaluation Plan
Program development is labor intensive and requires
plotting a strategy that will streamline the process. Below
is a checklist of questions to ask when formulating a model
of the program and establishing an evaluation plan.
Clarify Goals and Objectives
- Are goals and objectives defined in measurable terms?
- Does each goal and objective contain the four required elements (i.e. who, what, when, how much)?
- Are they directly linked to the intervention?
Create a Model of Your Program
- Does the model contain the following categories: intervention(s), target population, objectives and goals?
- Is every element in the model directly linked to
another element in the model?
Formulate Evaluation Questions
- Are your questions reflective of the program components?
- Can you gather the data needed to answer these
- Will the answers to the questions help those who will
use the results of your evaluation?
Determine What Type of Evaluation You Want to
- Did you choose the type of evaluation (process, outcome or impact) that will provide the information needed?
- Do you have the resources (time, money, etc.) to
conduct the type of evaluation selected?
Choose Data Collection Method(s)
- Will the data collection method you chose provide the data needed?
- Are your methods appropriate to the type of evaluation being conducted?
- Are your data collection tools reliable and
- Is your data analysis appropriate for the type of data collected?
- Will your analysis answer the evaluation
- Does your report fulfill the requirements given?
- Is your report understandable to the audience?
- Did you report your findings clearly?
There are three types of program evaluation: process,
outcome and impact. Each type provides different
information. In order to choose what type of evaluation
would be most appropriate, you must determine what questions
need to answered by the program. Below are explanations of
each type of evaluation as well as the question(s) that each
- Process (also known as accountability or monitoring)
addresses the way(s) which a program is implemented as
well as the conditions under which the program is taking
place. A process evaluation can also assess the materials
and activities that are being developed for content and
implementation appropriateness. Three questions that
should be asked when considering a process evaluation are:
- Is the program reaching its target population?
- Is the program being administered consistently and in accordance with the program's specifications?
- What resources will be needed to implement the
- Outcome is the most common type of evaluation
performed. It examines and measures the immediate effects
of a program on the target audience and determines whether
objectives were met. An outcome evaluation can only be
conducted if program objectives have been clearly defined
and stated in measureable terms. An outcome evaluation
seeks to answer two questions:
- Were there changes in the target audience based on the program's goals and objectives?
- Can these changes be attributed to the
- Impact is the most difficult type of evaluation to
perform due to the amount of time and resources necessary
to adequately assess the impact of a program. An impact
evaluation determines the effects of a program on its
long-term goals. It does not determine the effects of the
program reaching its objectives. Typically, an impact
evaluation answers one question:
- Did the program achieve its long term goals?
It is common to use a combination of evaluation types for
a program. Process and outcome evaluations are frequently
used to measure the effects of a program, however, only a
few programs include impact evaluations. If you are
uncertain about what type of evaluation you should use, two
questions to ask are:
- What are the available resources for the project? Limited resources may mean only doing a process evaluation.
- What information should be obtained? To find out
whether a program is effective, an outcome or impact
evaluation is needed.
Selecting A Data Collection Method(s)
Once goals and objectives have been established and the
right evaluation questions have been formulated, a method of
collecting data must be chosen that will best answer the
Choosing the appropriate data collection tool(s) will be
crucial to determining the effectiveness of the program.
Data collection tools must be designed to obtain the
information required by the evaluation questions. Before
deciding which method of data collection to use, the
following questions must be answered:
- What type of evaluation is being used (process, outcome, impact or a combination of these)?
- How much and what type of information is needed to answer the evaluation questions?
- Will there be data for all program components or just one or two?
- Do instruments exist to collect the data or do they need to be developed?
- How much time is there for data collection?
Data Collection Tools
- Questionnaires will measure participant's knowledge, attitudes, and/or traits.
- Interviews are used to obtain testimonials to how much
participants like the program or how they have changed while
participating in the program.
- Records/Files yield demographic information as well as other personal data.
- Observation provides information through direct observation of behaviors.
- Existing Data Collection Tools may provide a
cost-effective means for collecting data. If existing data
collection tools (questionnaires) are used, select tools
that demonstrate a high level of reliability (the ability
of the instrument to yield the same results on separate
occasions in the absence of changed behaviors, knowledge
or attitudes) and validity (the ability of the tool to
measure what it is supposed to measure). It is recommended
to use existing standardized instruments when possible
because the reliability and validity has been determined
through pilot testing. If you are using an instrument(s)
of your own design, they should be pilot tested to ensure
reliability and validity - a time consuming task.
Generally, data analysis techniques will be determined by
the evaluation questions and the methods used to collect
data. Data analysis can be as simple as calculating
percentages or as complex as performing a time series or
regression analysis. However, be aware that more complex
analysis usually requires assistance from someone who has
expertise in data analysis. Because specific aspects of
analysis are extremely complex, only three basic types of
data analysis are represented below.
- Descriptive Statistics simply describe the people who
participated in the program. It is important not to include
any statements regarding changes in participants, just the
facts. Example questions that can be answered by descriptive
- How many people participated in the program?
- What age group(s) were present in the program?
- What percentage of the participants were male or female?
- What percentage of the participants were from
- Correlational Statistics relate one variable to
another variable - they do not make a determination of cause
and effect. Example questions that can be answered by
correlational statistics include:
- Was there a relationship between teacher knowledge and workshop participation?
- Was there a relationship between number of years
teaching and the workshop choice?
- Tests of Statistical Significance makes the
determination of whether changes actually occurred and if
the changes were caused by the program. Example questions
that can be answered by tests of statistical significance
- Was there a change in knowledge after participation in the program?
- Was the change due to participation in the
The general rule is to begin with the least complex
analyses and work toward the most complex analysis technique
possible with the available expertise.
Reporting Evaluation Findings
Once data has been analyzed and interpreted, a report
will need to be prepared for purposes of disseminating and
publishing program results. Please remember that finding no
change among program participants is as important as finding
changes and both should be reported.
The standard framework to writing a good evaluation
- an executive summary which provides a brief overview of the evaluation;
- an introduction which describes the program, its
components, the target population, and the goals and
objectives of the program;
- a methods section which describes how the program was
actually implemented as well as how the data was collected,
what instruments were used to collect the data and how the
data was analyzed;
- the results section of the data analysis (it is
important to note here that this section should contain
concrete data, not interpretations);
- a discussion section which explains how the data was
interpreted, provides answers to evaluation questions,
discloses any problems encountered in the evaluation, and
suggests what could be done in the future to improve other
similar evaluations; and
- a recommendations section where recommendations are
made based on findings (this section is not always
Evaluation reports should be written so that they are
easily understandable to both lay people and professionals,
and formatted in a logical, attractive manner. Use the
following guidelines when writing the program final
- Do not over generalize your findings - if the program
was effective with pre-service science educators, do not
claim that the program will be effective with all
- Do not call modest changes or differences a success.
- Report total outcomes as well as partial
- Know the stakeholder that will use the results of your evaluation.
- Always be aware of the resources available to you.
- Design the evaluation to ask the right questions.
- Report results and findings in a clear and accurate manner.
- Keep your evaluation focused.
- When necessary, seek assistance from individuals with the related expertise that is needed.
- Develop a plan for documenting and maintaining all evaluation activities and data.
- A well done process evaluation provides valuable
information, and may be more appropriate for the scope of
your program than an outcome or impact assessment.
- Babbie, E.R. Survey research
methods. Belmont, CA: Wadsworth, 1973.
- Bogdan, R.C., and Biklen,
S.K. Qualitative research in
education. 2nd ed. Newbury Park, Calif.: Sage,
- Borg, W.R., and Gall, M.D. Exploring relationships
between variables: The causal-comparative method. In
Educational research: An introduction. 5th ed. New York:
- Campbell, D.T., and Stanley, J.C. Experimental and
quasiexperimental designs for research. Chicago: Rand
- Campbell, J.P., Daft, R.L., and Hulin, C.L. What to
study: Generating and developing research questions. Newbury
Park, Calif.: Sage, 1982.
- Carey, L. Measuring and evaluating school learning.
Boston: Allyn & Bacon, 1988.
- Cook, T.D., and Campbell, D.T. Quasiexperimentation:
Designs and analysis issues for field settings. Chicago:
Rand McNally, 1979.
- Fink, A., and Kosecoff, J. How to conduct surveys: A
step-by-step guide. Newbury Park, Calif.: Sage, 1985.
- Fraenkel, J.R., and Wallen, N.E. How to Design and
Evaluate Research in Education. 3rd ed. McGraw-Hill, Inc.,
1996. (This book has an excellent topical
- Hatry, H.P., Newcomer, K.E., and Wholey, J.S. Handbook
of Practical Program Evaluation. San Francisco: Jossey-Bass,
- Hopkins, D. A teacher's guide to classroom research.
Philadelphia, PA: Open University Press, 1985.
- Kenny, D.A. Correlation and causality. New York:
- Krippendorff, K. Content analysis: An introduction to
its methodology. Newbury Park, Calif.: Sage, 1980.
- Lancy, D.F. Qualitative research in education: An
introduction to the major traditions. New York: Longman,
- Leibetrau, A.M. Measures of Association. Newbury Park,
Calif.: Sage, 1979.
- Phillips, D.C. Towards an evaluation of the experiment
in educational contexts. Educational Researcher, 10(6):
- Shulman, L.S. Disciplines of inquiry in education: An
overview. In R.M. Jaeger (Ed.) Complementary methods for
research in education. Washington, D.C.: American
Educational Research Association, 1988.
- Tuckman, B.W. Identifying and labeling variables. In
Conducting educational research. 4th ed. San Diego, Calif.:
Harcourt, Brace, Jovanovich, 1994.
- Weber, R.P. Basic content analysis. 2nd ed. Newbury
Park, Calif.: Sage, 1990.
- Wholey, J.S., Hatry, H.P., and Newcomer, K.E. Handbook
of Practical Program Evaluation. San Francisco, Calif.:
Jossey-Bass, Inc., 1994.
- Wilson, S. The use of ethnographic techniques in
educational research. Review of Educational Research, 47:
- Cochran, W.G. Sampling techniques. 3rd ed. New York:
- Kish, L. Survey Sampling New York: Wiley, 1965.
- Converse, J.M., and Presser, S. Survey Questions:
Handicrafting the Standardized Questionnaire. Newbury Park,
Calif.: Sage, 1976.
- Fink, A., and Kosecoff, J. How to Conduct Surveys: A
Step by Step Guide. Newbury Park, Calif.: Sage, 1985.
- Sudman, S., and Bradburn, N.M. Asking Questions: A
Practical Guide to Questionnaire Design. San Francisco:
Validity & Reliability
- Brinberg, S., and McGrath, J.E. Validity and the
research process. Newbury Park, Calif.: Sage, 1985.
- Carmines, E.G., and Zeller, R.A. Reliability and
validity assessment. Newbury Park, Calif.: Sage,
- Bruning, J.L., and Kintz, B.L. Computational handbook
of statistics. 3rd ed. Glenview, IL: Scott, Foresman,
- Conover, W.J. Practical nonparametric statistics. New
York: Wiley, 1971.
- Schutte, J.G. Everything you always wanted to know
about elementary statistics (but were afraid to ask).
Englewood Cliffs, NJ: Prentice-Hall, 1977.
1998-2013 The Association of Universities for Research in Astronomy, Inc.
All Rights Reserved.
Questions or comments, please send email to: