logo
Introduction
Call for Proposals
Submittal
Resources
Archives
Education
STScI
NASA


STScI Logo
 

HST Cycle 8 E/PO
Resources

ELEMENTS OF A PROGRAM EVALUATION

What Is Program Evaluation?

By definition, program evaluation is "the systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of social intervention programs." In practice, program evaluation plays a significant role in program development and assessment. From concept to planning, application to results, the systematic evaluation of each step of a program will serve as a mechanism to developing a realistic program that is clear, comprehensive and measurable. A solid program evaluation will also expedite the dissemination and publication process. Some key areas where incorporating program evaluation methods could be beneficial include:

  • new program designs and development;
  • program management and tracking;
  • efficiency of program implementation;
  • accountability;
  • dissemination; and
  • program effectiveness.

UP

Developing An Evaluation Plan

Program development is labor intensive and requires plotting a strategy that will streamline the process. Below is a checklist of questions to ask when formulating a model of the program and establishing an evaluation plan.

Clarify Goals and Objectives
  • Are goals and objectives defined in measurable terms?
  • Does each goal and objective contain the four required elements (i.e. who, what, when, how much)?
  • Are they directly linked to the intervention?
Create a Model of Your Program
  • Does the model contain the following categories: intervention(s), target population, objectives and goals?
  • Is every element in the model directly linked to another element in the model?
Formulate Evaluation Questions
  • Are your questions reflective of the program components?
  • Can you gather the data needed to answer these questions?
  • Will the answers to the questions help those who will use the results of your evaluation?
Determine What Type of Evaluation You Want to Conduct
  • Did you choose the type of evaluation (process, outcome or impact) that will provide the information needed?
  • Do you have the resources (time, money, etc.) to conduct the type of evaluation selected?
Choose Data Collection Method(s)
  • Will the data collection method you chose provide the data needed?
  • Are your methods appropriate to the type of evaluation being conducted?
  • Are your data collection tools reliable and valid?
Analyze Data
  • Is your data analysis appropriate for the type of data collected?
  • Will your analysis answer the evaluation questions?
Reporting Findings
  • Does your report fulfill the requirements given?
  • Is your report understandable to the audience?
  • Did you report your findings clearly?

UP

Evaluation Types

There are three types of program evaluation: process, outcome and impact. Each type provides different information. In order to choose what type of evaluation would be most appropriate, you must determine what questions need to answered by the program. Below are explanations of each type of evaluation as well as the question(s) that each should answer.

  • Process (also known as accountability or monitoring) addresses the way(s) which a program is implemented as well as the conditions under which the program is taking place. A process evaluation can also assess the materials and activities that are being developed for content and implementation appropriateness. Three questions that should be asked when considering a process evaluation are:
    1. Is the program reaching its target population?
    2. Is the program being administered consistently and in accordance with the program's specifications?
    3. What resources will be needed to implement the program?
  • Outcome is the most common type of evaluation performed. It examines and measures the immediate effects of a program on the target audience and determines whether objectives were met. An outcome evaluation can only be conducted if program objectives have been clearly defined and stated in measureable terms. An outcome evaluation seeks to answer two questions:
    1. Were there changes in the target audience based on the program's goals and objectives?
    2. Can these changes be attributed to the program?
  • Impact is the most difficult type of evaluation to perform due to the amount of time and resources necessary to adequately assess the impact of a program. An impact evaluation determines the effects of a program on its long-term goals. It does not determine the effects of the program reaching its objectives. Typically, an impact evaluation answers one question:
    1. Did the program achieve its long term goals?

It is common to use a combination of evaluation types for a program. Process and outcome evaluations are frequently used to measure the effects of a program, however, only a few programs include impact evaluations. If you are uncertain about what type of evaluation you should use, two questions to ask are:

  • What are the available resources for the project? Limited resources may mean only doing a process evaluation.
  • What information should be obtained? To find out whether a program is effective, an outcome or impact evaluation is needed.

UP

Selecting A Data Collection Method(s)

Once goals and objectives have been established and the right evaluation questions have been formulated, a method of collecting data must be chosen that will best answer the questions.

Choosing the appropriate data collection tool(s) will be crucial to determining the effectiveness of the program. Data collection tools must be designed to obtain the information required by the evaluation questions. Before deciding which method of data collection to use, the following questions must be answered:

  • What type of evaluation is being used (process, outcome, impact or a combination of these)?
  • How much and what type of information is needed to answer the evaluation questions?
  • Will there be data for all program components or just one or two?
  • Do instruments exist to collect the data or do they need to be developed?
  • How much time is there for data collection?
Data Collection Tools
  • Questionnaires will measure participant's knowledge, attitudes, and/or traits.
  • Interviews are used to obtain testimonials to how much participants like the program or how they have changed while participating in the program.
  • Records/Files yield demographic information as well as other personal data.
  • Observation provides information through direct observation of behaviors.
  • Existing Data Collection Tools may provide a cost-effective means for collecting data. If existing data collection tools (questionnaires) are used, select tools that demonstrate a high level of reliability (the ability of the instrument to yield the same results on separate occasions in the absence of changed behaviors, knowledge or attitudes) and validity (the ability of the tool to measure what it is supposed to measure). It is recommended to use existing standardized instruments when possible because the reliability and validity has been determined through pilot testing. If you are using an instrument(s) of your own design, they should be pilot tested to ensure reliability and validity - a time consuming task.

UP

Analyzing Data

Generally, data analysis techniques will be determined by the evaluation questions and the methods used to collect data. Data analysis can be as simple as calculating percentages or as complex as performing a time series or regression analysis. However, be aware that more complex analysis usually requires assistance from someone who has expertise in data analysis. Because specific aspects of analysis are extremely complex, only three basic types of data analysis are represented below.

  • Descriptive Statistics simply describe the people who participated in the program. It is important not to include any statements regarding changes in participants, just the facts. Example questions that can be answered by descriptive statistics include:
    1. How many people participated in the program?
    2. What age group(s) were present in the program?
    3. What percentage of the participants were male or female?
    4. What percentage of the participants were from underserved/underrepresented groups?
  • Correlational Statistics relate one variable to another variable - they do not make a determination of cause and effect. Example questions that can be answered by correlational statistics include:
    1. Was there a relationship between teacher knowledge and workshop participation?
    2. Was there a relationship between number of years teaching and the workshop choice?
  • Tests of Statistical Significance makes the determination of whether changes actually occurred and if the changes were caused by the program. Example questions that can be answered by tests of statistical significance include:
    1. Was there a change in knowledge after participation in the program?
    2. Was the change due to participation in the program?

The general rule is to begin with the least complex analyses and work toward the most complex analysis technique possible with the available expertise.

UP

Reporting Evaluation Findings

Once data has been analyzed and interpreted, a report will need to be prepared for purposes of disseminating and publishing program results. Please remember that finding no change among program participants is as important as finding changes and both should be reported.

The standard framework to writing a good evaluation report is:

  • an executive summary which provides a brief overview of the evaluation;
  • an introduction which describes the program, its components, the target population, and the goals and objectives of the program;
  • a methods section which describes how the program was actually implemented as well as how the data was collected, what instruments were used to collect the data and how the data was analyzed;
  • the results section of the data analysis (it is important to note here that this section should contain concrete data, not interpretations);
  • a discussion section which explains how the data was interpreted, provides answers to evaluation questions, discloses any problems encountered in the evaluation, and suggests what could be done in the future to improve other similar evaluations; and
  • a recommendations section where recommendations are made based on findings (this section is not always necessary).

Evaluation reports should be written so that they are easily understandable to both lay people and professionals, and formatted in a logical, attractive manner. Use the following guidelines when writing the program final report:

  • Do not over generalize your findings - if the program was effective with pre-service science educators, do not claim that the program will be effective with all educators.
  • Do not call modest changes or differences a success.
  • Report total outcomes as well as partial outcomes.

UP

Evaluation Guidelines

  1. Know the stakeholder that will use the results of your evaluation.
  2. Always be aware of the resources available to you.
  3. Design the evaluation to ask the right questions.
  4. Report results and findings in a clear and accurate manner.
  5. Keep your evaluation focused.
  6. When necessary, seek assistance from individuals with the related expertise that is needed.
  7. Develop a plan for documenting and maintaining all evaluation activities and data.
  8. A well done process evaluation provides valuable information, and may be more appropriate for the scope of your program than an outcome or impact assessment.

UP

Bibliography

Program Evaluation/Research
  • Babbie, E.R. Survey research methods. Belmont, CA: Wadsworth, 1973.
  • Bogdan, R.C., and Biklen, S.K. Qualitative research in education. 2nd ed. Newbury Park, Calif.: Sage, 1992.
  • Borg, W.R., and Gall, M.D. Exploring relationships between variables: The causal-comparative method. In Educational research: An introduction. 5th ed. New York: Longman, 1989.
  • Campbell, D.T., and Stanley, J.C. Experimental and quasiexperimental designs for research. Chicago: Rand McNally, 1963.
  • Campbell, J.P., Daft, R.L., and Hulin, C.L. What to study: Generating and developing research questions. Newbury Park, Calif.: Sage, 1982.
  • Carey, L. Measuring and evaluating school learning. Boston: Allyn & Bacon, 1988.
  • Cook, T.D., and Campbell, D.T. Quasiexperimentation: Designs and analysis issues for field settings. Chicago: Rand McNally, 1979.
  • Fink, A., and Kosecoff, J. How to conduct surveys: A step-by-step guide. Newbury Park, Calif.: Sage, 1985.
  • Fraenkel, J.R., and Wallen, N.E. How to Design and Evaluate Research in Education. 3rd ed. McGraw-Hill, Inc., 1996. (This book has an excellent topical bibliography.)
  • Hatry, H.P., Newcomer, K.E., and Wholey, J.S. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass, 1994.
  • Hopkins, D. A teacher's guide to classroom research. Philadelphia, PA: Open University Press, 1985.
  • Kenny, D.A. Correlation and causality. New York: Wiley, 1979.
  • Krippendorff, K. Content analysis: An introduction to its methodology. Newbury Park, Calif.: Sage, 1980.
  • Lancy, D.F. Qualitative research in education: An introduction to the major traditions. New York: Longman, 1993.
  • Leibetrau, A.M. Measures of Association. Newbury Park, Calif.: Sage, 1979.
  • Phillips, D.C. Towards an evaluation of the experiment in educational contexts. Educational Researcher, 10(6): 13-20.
  • Shulman, L.S. Disciplines of inquiry in education: An overview. In R.M. Jaeger (Ed.) Complementary methods for research in education. Washington, D.C.: American Educational Research Association, 1988.
  • Tuckman, B.W. Identifying and labeling variables. In Conducting educational research. 4th ed. San Diego, Calif.: Harcourt, Brace, Jovanovich, 1994.
  • Weber, R.P. Basic content analysis. 2nd ed. Newbury Park, Calif.: Sage, 1990.
  • Wholey, J.S., Hatry, H.P., and Newcomer, K.E. Handbook of Practical Program Evaluation. San Francisco, Calif.: Jossey-Bass, Inc., 1994.
  • Wilson, S. The use of ethnographic techniques in educational research. Review of Educational Research, 47: 245-265.
Sampling
  • Cochran, W.G. Sampling techniques. 3rd ed. New York: Wiley, 1977.
  • Kish, L. Survey Sampling New York: Wiley, 1965.
Questionnaire/Survey Development
  • Converse, J.M., and Presser, S. Survey Questions: Handicrafting the Standardized Questionnaire. Newbury Park, Calif.: Sage, 1976.
  • Fink, A., and Kosecoff, J. How to Conduct Surveys: A Step by Step Guide. Newbury Park, Calif.: Sage, 1985.
  • Sudman, S., and Bradburn, N.M. Asking Questions: A Practical Guide to Questionnaire Design. San Francisco: Jossey-Bass, 1982.
Validity & Reliability
  • Brinberg, S., and McGrath, J.E. Validity and the research process. Newbury Park, Calif.: Sage, 1985.
  • Carmines, E.G., and Zeller, R.A. Reliability and validity assessment. Newbury Park, Calif.: Sage, 1979.
Statistics
  • Bruning, J.L., and Kintz, B.L. Computational handbook of statistics. 3rd ed. Glenview, IL: Scott, Foresman, 1987.
  • Conover, W.J. Practical nonparametric statistics. New York: Wiley, 1971.
  • Schutte, J.G. Everything you always wanted to know about elementary statistics (but were afraid to ask). Englewood Cliffs, NJ: Prentice-Hall, 1977.

UP


Copyright © 1998-2013 The Association of Universities for Research in Astronomy, Inc.
All Rights Reserved.
Questions or comments, please send email to: cyclehstepo@stsci.edu