Elements of a Program Evaluation
By definition, program evaluation is "the systematic application of social research procedures for assessing
the conceptualization, design, implementation, and utility of social intervention programs." In practice, program
evaluation plays a significant role in program development and assessment. From concept to planning, application to
results, the systematic evaluation of each step of a program will serve as a mechanism to developing a
realistic program that is clear, comprehensive and measurable. A solid program evaluation will also expedite
the dissemination and publication process. Some key areas where incorporating program evaluation methods could be
- new program designs and development;
- program management and tracking;
- efficiency of program implementation;
- dissemination; and
- program effectiveness.
Program development is labor intensive and requires plotting a strategy that will streamline the process. Below
is a checklist of questions to ask when formulating a model of the program and establishing an evaluation plan.
Clarify Goals and Objectives
- Are goals and objectives defined in measurable terms?
- Does each goal and objective contain the four required elements (i.e. who, what, when, how much)?
- Are they directly linked to the intervention?
Create a Model of Your Program
- Does the model contain the following categories: intervention(s), target population, objectives and goals?
- Is every element in the model directly linked to another element in the model?
Formulate Evaluation Questions
- Are your questions reflective of the program components?
- Can you gather the data needed to answer these questions?
- Will the answers to the questions help those who will use the results of your evaluation?
Determine What Type of Evaluation You Want to Conduct
- Did you choose the type of evaluation (process, outcome or impact) that will provide the information needed?
- Do you have the resources (time, money, etc.) to conduct the type of evaluation selected?
Choose Data Collection Method(s)
- Will the data collection method you chose provide the data needed?
- Are your methods appropriate to the type of evaluation being conducted?
- Are your data collection tools reliable and valid?
- Is your data analysis appropriate for the type of data collected?
- Will your analysis answer the evaluation questions?
- Does your report fulfill the requirements given?
- Is your report understandable to the audience?
- Did you report your findings clearly?
There are three types of program evaluation: process, outcome and impact. Each type provides different
information. In order to choose what type of evaluation would be most appropriate, you must determine what questions
need to answered by the program. Below are explanations of each type of evaluation as well as the question(s) that each
Process (also known as accountability or monitoring) addresses the way(s) which a program is implemented as
well as the conditions under which the program is taking place. A process evaluation can also assess the materials
and activities that are being developed for content and implementation appropriateness. Three questions that
should be asked when considering a process evaluation are:
- Is the program reaching its target population?
- Is the program being administered consistently and in accordance with the program's specifications?
- What resources will be needed to implement the program?
Outcome is the most common type of evaluation performed. It examines and measures the immediate effects
of a program on the target audience and determines whether objectives were met. An outcome evaluation can only be
conducted if program objectives have been clearly defined and stated in measureable terms. An outcome evaluation
seeks to answer two questions:
- Were there changes in the target audience based on the program's goals and objectives?
- Can these changes be attributed to the program?
Impact is the most difficult type of evaluation to perform due to the amount of time and resources necessary
to adequately assess the impact of a program. An impact evaluation determines the effects of a program on its
long-term goals. It does not determine the effects of the program reaching its objectives. Typically, an impact
evaluation answers one question:
- Did the program achieve its long term goals?
It is common to use a combination of evaluation types for a program. Process and outcome evaluations are frequently
used to measure the effects of a program, however, only a few programs include impact evaluations. If you are
uncertain about what type of evaluation you should use, two questions to ask are:
- What are the available resources for the project? Limited resources may mean only doing a process evaluation.
What information should be obtained? To find out whether a program is effective, an outcome or impact
evaluation is needed.
Once goals and objectives have been established and the right evaluation questions have been formulated, a method of
collecting data must be chosen that will best answer the questions.
Choosing the appropriate data collection tool(s) will be crucial to determining the effectiveness of the program.
Data collection tools must be designed to obtain the information required by the evaluation questions. Before
deciding which method of data collection to use, the following questions must be answered:
- What type of evaluation is being used (process, outcome, impact or a combination of these)?
- How much and what type of information is needed to answer the evaluation questions?
- Will there be data for all program components or just one or two?
- Do instruments exist to collect the data or do they need to be developed?
- How much time is there for data collection?
Data Collection Tools
- Questionnaires will measure participant's knowledge, attitudes, and/or traits.
Interviews are used to obtain testimonials to how much participants like the program or how they have
changed while participating in the program.
- Records/Files yield demographic information as well as other personal data.
- Observation provides information through direct observation of behaviors.
Existing Data Collection Tools may provide a cost-effective means for collecting data. If existing data
collection tools (questionnaires) are used, select tools that demonstrate a high level of reliability (the ability
of the instrument to yield the same results on separate occasions in the absence of changed behaviors, knowledge
or attitudes) and validity (the ability of the tool to measure what it is supposed to measure). It is recommended
to use existing standardized instruments when possible because the reliability and validity has been determined
through pilot testing. If you are using an instrument(s) of your own design, they should be pilot tested to ensure
reliability and validity - a time consuming task.
Generally, data analysis techniques will be determined by the evaluation questions and the methods used to collect
data. Data analysis can be as simple as calculating percentages or as complex as performing a time series or
regression analysis. However, be aware that more complex analysis usually requires assistance from someone who has
expertise in data analysis. Because specific aspects of analysis are extremely complex, only three basic types of
data analysis are represented below.
Descriptive Statistics simply describe the people who participated in the program. It is important not to include
any statements regarding changes in participants, just the facts. Example questions that can be answered by descriptive
- How many people participated in the program?
- What age group(s) were present in the program?
- What percentage of the participants were male or female?
- What percentage of the participants were from underserved/underrepresented groups?
Correlational Statistics relate one variable to another variable - they do not make a determination of cause
and effect. Example questions that can be answered by correlational statistics include:
- Was there a relationship between teacher knowledge and workshop participation?
- Was there a relationship between number of years teaching and the workshop choice?
Tests of Statistical Significance makes the determination of whether changes actually occurred and if
the changes were caused by the program. Example questions that can be answered by tests of statistical significance
- Was there a change in knowledge after participation in the program?
- Was the change due to participation in the program?
The general rule is to begin with the least complex analyses and work toward the most complex analysis technique
possible with the available expertise.
Once data has been analyzed and interpreted, a report will need to be prepared for purposes of disseminating and
publishing program results. Please remember that finding no change among program participants is as important as finding
changes and both should be reported.
The standard framework to writing a good evaluation report is:
- an executive summary which provides a brief overview of the evaluation;
an introduction which describes the program, its components, the target population, and the goals and
objectives of the program;
a methods section which describes how the program was actually implemented as well as how the data was collected,
what instruments were used to collect the data and how the data was analyzed;
the results section of the data analysis (it is important to note here that this section should contain
concrete data, not interpretations);
a discussion section which explains how the data was interpreted, provides answers to evaluation questions,
discloses any problems encountered in the evaluation, and suggests what could be done in the future to improve other
similar evaluations; and
a recommendations section where recommendations are made based on findings (this section is not always
Evaluation reports should be written so that they are easily understandable to both lay people and professionals,
and formatted in a logical, attractive manner. Use the following guidelines when writing the program final
Do not over generalize your findings - if the program was effective with pre-service science educators, do not
claim that the program will be effective with all educators.
- Do not call modest changes or differences a success.
- Report total outcomes as well as partial outcomes.
- Know the stakeholder that will use the results of your evaluation.
- Always be aware of the resources available to you.
- Design the evaluation to ask the right questions.
- Report results and findings in a clear and accurate manner.
- Keep your evaluation focused.
- When necessary, seek assistance from individuals with the related expertise that is needed.
- Develop a plan for documenting and maintaining all evaluation activities and data.
- A well done process evaluation provides valuable information, and may be more appropriate for the scope of
your program than an outcome or impact assessment.
- Babbie, E.R. Survey research methods. Belmont, CA: Wadsworth, 1973.
- Bogdan, R.C., and Biklen, S.K. Qualitative research in education. 2nd ed. Newbury Park, Calif.: Sage, 1992.
Borg, W.R., and Gall, M.D. Exploring relationships between variables: The causal-comparative method. In
Educational research: An introduction. 5th ed. New York: Longman, 1989.
Campbell, D.T., and Stanley, J.C. Experimental and quasiexperimental designs for research. Chicago: Rand McNally, 1963.
Campbell, J.P., Daft, R.L., and Hulin, C.L. What to study: Generating and developing research questions. Newbury Park,
Calif.: Sage, 1982.
- Carey, L. Measuring and evaluating school learning. Boston: Allyn & Bacon, 1988.
Cook, T.D., and Campbell, D.T. Quasiexperimentation: Designs and analysis issues for field settings. Chicago: Rand
- Fink, A., and Kosecoff, J. How to conduct surveys: A step-by-step guide. Newbury Park, Calif.: Sage, 1985.
Fraenkel, J.R., and Wallen, N.E. How to Design and Evaluate Research in Education. 3rd ed. McGraw-Hill, Inc., 1996.
(This book has an excellent topical bibliography.)
Hatry, H.P., Newcomer, K.E., and Wholey, J.S. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass, 1994.
- Hopkins, D. A teacher's guide to classroom research. Philadelphia, PA: Open University Press, 1985.
- Kenny, D.A. Correlation and causality. New York: Wiley, 1979.
- Krippendorff, K. Content analysis: An introduction to its methodology. Newbury Park, Calif.: Sage, 1980.
- Lancy, D.F. Qualitative research in education: An introduction to the major traditions. New York: Longman, 1993.
- Leibetrau, A.M. Measures of Association. Newbury Park, Calif.: Sage, 1979.
Phillips, D.C. Towards an evaluation of the experiment in educational contexts. Educational Researcher, 10(6): 13-20.
Shulman, L.S. Disciplines of inquiry in education: An overview. In R.M. Jaeger (Ed.) Complementary methods for research in
education. Washington, D.C.: American Educational Research Association, 1988.
Tuckman, B.W. Identifying and labeling variables. In Conducting educational research. 4th ed. San Diego, Calif.:
Harcourt, Brace, Jovanovich, 1994.
- Weber, R.P. Basic content analysis. 2nd ed. Newbury Park, Calif.: Sage, 1990.
Wholey, J.S., Hatry, H.P., and Newcomer, K.E. Handbook of Practical Program Evaluation. San Francisco, Calif.:
Jossey-Bass, Inc., 1994.
Wilson, S. The use of ethnographic techniques in educational research. Review of Educational Research, 47: 245-265.
- Cochran, W.G. Sampling techniques. 3rd ed. New York: Wiley, 1977.
- Kish, L. Survey Sampling New York: Wiley, 1965.
Converse, J.M., and Presser, S. Survey Questions: Handicrafting the Standardized Questionnaire. Newbury Park, Calif.:
- Fink, A., and Kosecoff, J. How to Conduct Surveys: A Step by Step Guide. Newbury Park, Calif.: Sage, 1985.
Sudman, S., and Bradburn, N.M. Asking Questions: A Practical Guide to Questionnaire Design. San Francisco: Jossey-Bass,
Validity & Reliability
- Brinberg, S., and McGrath, J.E. Validity and the research process. Newbury Park, Calif.: Sage, 1985.
- Carmines, E.G., and Zeller, R.A. Reliability and validity assessment. Newbury Park, Calif.: Sage, 1979.
Bruning, J.L., and Kintz, B.L. Computational handbook of statistics. 3rd ed. Glenview, IL: Scott, Foresman, 1987.
- Conover, W.J. Practical nonparametric statistics. New York: Wiley, 1971.
Schutte, J.G. Everything you always wanted to know about elementary statistics (but were afraid to ask). Englewood
Cliffs, NJ: Prentice-Hall, 1977.