ELEMENTS OF A PROGRAM EVALUATION

WHAT IS PROGRAM EVALUATION?

• new program designs and development;
• program management and tracking;
• efficiency of program implementation;
• accountability;
• dissemination; and
• program effectiveness.

EVALUATION TYPES

• Process (also know as accountability or monitoring) addresses the way(s) which a program is implemented as well as the conditions under which the program is taking place. A process evaluation can also assess the materials and activities that are being developed for content and implementation appropriateness. Three questions that should be asked when considering a process evaluation are:
  
  1. Is the program reaching its target population?
  2. Is the program being administered consistently and in accordance with the program's specifications?
  3. What resources will be needed to implement the program?

• Outcome is the most common type of evaluation performed. It examines and measures the immediate effects of a program on the target audience and determines whether objectives were met. An outcome evaluation can only be conducted if program objectives have been clearly defined and stated in measureable terms. An outcome evaluation seeks to answer two questions:
  
  1. Were there changes in the target audience based on the program's goals and objectives?
  2. Can these changes be attributed to the program?

• Impact is the most difficult type of evaluation to perform due to the amount of time and resources necessary to adequately assess the impact of a program. An impact evaluation determines the effects of a program on its long-term goals. It does not determine the effects of the program reaching its objectives. Typically, an impact evaluation answers one question:
  
  1. Did the program achieve its long term goals?

• What are the available resources for the project? Limited resources may mean only doing a process evaluation.
• What information should be obtained? To find out whether a program is effective, an outcome or impact evaluation is needed.

UP

CHOOSING A METHODOLOGY

• Case Study which examines changes in a single group.
• Time Series which investigates changes to a group over time. In this type of evaluation, data must be collected at intervals over a period of time.
• Comparison Group which compares a program's participants to a similar group (control group) that does not participate in the program (either by receiving no services or participating in another program).

The ideal method to use is a comparison group where data is collected from two identical groups at the start of a program as well as several times throughout the implementation of the program. This is also the most difficult method to use and requires extensive resources.

Questions that should be asked when determining what methodology design to use are:

• Do I want data for only one point in time or data that compares participants both before and after participation in the program?
• Do I want to compare the program participants to a group who does not participate in the program?
• Do I have access to a comparison group who will not participate in the program?
• What expertise do I have or have access to for analyzing the data collected?
• How much time and money is available for data collection?

CHOOSING A DATA COLLECTION METHOD(S)

• What type of evaluation is being used (process, outcome, impact or a combination of these)?
• How much and what type of information is needed to answer the evaluation questions?
• Will there be data for all program components or just one or two?
• Do instruments exist to collect the data or do they need to be developed?
• How much time is there for data collection?

Data Collection Tools

• Questionnaires will measure participant's traits (i.e. knowledge, attitudes, etc. )
• Interviews are used to obtain testimonials to how much participants like the program or how they have changed while participating in the program.
• Records/Files yield demographic information as well as other personal data.
• Observation provides information through direct observation of behaviors.
• Existing Data/Statistics for statiscial information. If existing data collection tools (questionnaires) are used, look for ones that demonstrate a high level of reliability (the ability of the instrument to yield the same results on separate occassions in the absence of changed behaviors, knowledge or attitudes) and validity (the ability of the tool to measure what it is suppose to measure). It is recommended to use existing standardized instruments when possible because the reliability and validity has been determined through pilot testing. If you are using an instrument(s) of your own design, they should be pilot tested to ensure reliability and validity - a time consuming task.
  Results will be most conclusive when outcomes are measured using more than one instrument or method.
  

ANALYZING DATA

• Descriptive Statistics simply describe the people who participated in the program. It is important not to include any statements regarding changes in participants, just the facts. Example questions that can be answered by descriptive statistics include:
  
  1. How many people participated in the program?
  2. What age group(s) were present in the program?
  3. What percentage of the participants were male or female?
  4. What percentage of the participants were from underserved/underrepresented groups?

• Correlational Statistics relate one variable to another variable - they do not make a determination of cause and effect. Example questions that can be answered by correlational statistics include:
  
  1. Was there a relationship between teacher knowledge and workshop participation?
  2. Was there a relationship between number of years teaching and the workshop choice?

• Tests of Statistical Significance makes the detemination of whether changes actually occurred and if the changes were caused by the program. Example questions that can be answered by tests of statistical significance include:
  
  1. Was there a change in knowledge after participation in the program?
  2. Was the change due to participation in the program?

REPORTING EVALUATION FINDINGS

• an executive summary which provides a brief overview of the evaluation;
• an introduction which describes the program, its components, the target population, and the goals and objectives of the program;
• a methodology which describes how the program was actually implemented as well as how the data was collected, what instruments were used to collect the data and how the data was analyzed;
• the results section of the data analysis (it is important to note here that this section should contain concrete data, not interpretations);
• a discussion section which explains how the data was interpreted, provides answers to evaluation questions, discloses any problems encountered in the evaluation, and suggests what could be done in the future to improve other similar evaluations; and
• a recommendations section where recommendations are made based on findings (this section is not always necessary).

• Do not overgeneralize your findings - if the program was effective with pre-service science teachers, do not claim that the program will be effective with all teachers.
• Do not call modest changes or differences a success.
• Report total outcomes as well as partial outcomes.
• Be candid but do not be tactless.

DEVELOPING AN EVALUATION PLAN

• Are goals and objectives defined in measurable terms?
• Does each goal and objective contain the four required elements (i.e. who, what, when, how much)?
• Are they directly linked to the intervention?

• Does the model contain the following categories: intervention(s), target population, objectives and goals?
• Is every element in the model directly linked to another element in the model?

• Are your questions reflective of the program components?
• Can you gather the data needed to answer these questions?
• Will the answers to the questions help those who will use the results of your evaluation?

• Did you choose the type of evaluation (process, outcome or impact) that will provide the information needed?
• Do you have the resources (time, money, etc.) to conduct the type of evaluation choosen?

• Will the methodology you chose provide the answers to the evaluation questions?
• Do you have the resources to utilize the methodology choosen?
• Can you possibly use a more rigorous method?

• Will the data collection method you chose provide the data needed?
• Are your methods appropriate to the type of evaluation being conducted?
• Are your data collection tools reliable and valid?

• Is your data analysis appropriate for the type of data collected?
• Will your analysis answer the evaluation questions?

• Does your report fulfill the requirements given?
• Is your report understandable to the audience?
• Did you report your findings clearly?

GUIDELINES OF EVALUATION

1. Know who will use the results of your evaluation.
2. Always be aware of the resources available to you.
3. Make sure you ask the right questions.
4. If at all possible, use a control group.
5. Report what you find and don't report what you don't find.
6. Keep your evaluation focused.
7. Get help from experts when you need it.
8. Be flexible.
9. Document everything you do.
10. A well done process evaluation is better than a poorly done outcome/impact assessment.

BIBLIOGRAPHY

• Babbie, E.R. Survey research methods. Belmont, CA: Wadsworth, 1973.
• Bogdan, R.C., and Biklen, S.K. Qualitative research in education. 2nd ed. Newbury Park, Calif.: Sage, 1992.
• Borg, W.R., and Gall, M.D. Exploring relationships between variables: The causal-comparative method. In Educational research: An introduction. 5th ed. New York: Longman, 1989.
• Campbell, D.T., and Stanley, J.C. Experimental and quasiexperimental designs for research. Chicago: Rand McNally, 1963.
• Campbell, J.P., Daft, R.L., and Hulin, C.L. What to study: Generating and developing research questions. Newbury Park, Calif.: Sage, 1982.
• Carey, L. Measuring and evaluating school learning. Boston: Allyn & Bacon, 1988.
• Cook, T.D., and Campbell, D.T. Quasiexperimentation: Designs and analysis issues for field settings.. Chicago: Rand McNally, 1979.
• Fink, A., and Kosecoff, J. How to conduct surveys: A step-by-step guide. Newbury Park, Calif.: Sage, 1985.
• Fraenkel, J.R., and Wallen, N.E. How to Design and Evaluate Research in Education. 3rd ed. McGraw-Hill, Inc., 1996. (This book has an excellent topical bibliography.)
• Hatry, H.P., Newcomer, K.E., and Wholey, J.S. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass, 1994.
• Hopkins, D. A teacher's guide to classroom research. Philadelphia, PA: Open University Press, 1985.
• Kenny, D.A. Correlation and causality. New York: Wiley, 1979.
• Krippendorff, K. Content analysis: An introduction to its methodology. Newbury Park, Calif.: Sage, 1980.
• Lancy, D.F. Qualitative research in education: An introduction to the major traditions. New York: Longman, 1993.
• Leibetrau, A.M. Measures of Association. Newbury Park, Calif.: Sage, 1979.
• Phillips, D.C. Towards an evlauation of the experiment in educational contexts. Educational Researcher, 10(6): 13-20.
• Shulman, L.S. Disciplines of inquiry in education: An overview. In R.M. Jaeger (Ed.) Complementary methods for research in education. Washington, D.C.: American Educational Research Association, 1988.
• Tuckman, B.W. Identifying and labeling variables. In Conducting educational research. 4th ed. San Diego, Calif.: Harcourt, Brace, Jovanovich, 1994.
• Weber, R.P. Basic content analysis. 2nd ed. Newbury Park, Calif.: Sage, 1990.
• Wholey, J.S., Hatry, H.P., and Newcomer, K.E. Handbook of Practical Program Evaluation. San Francisco, Calif.: Jossey-Bass, Inc., 1994.
• Wilson, S. The use of ethnographic techniques in educational research. Review of Educational Research, 47: 245-265.

• Cochran, W.G. Sampling techniques. 3rd ed. New York: Wiley, 1977.
• Kish, L. Survey Sampling New York: Wiley, 1965.

• Converse, J.M., and Presser, S. Survey Questions: Handicrafting the Standardized Questionnaire. Newbury Park, Calif.: Sage, 1976.
• Fink, A., and Kosecoff, J. How to Conduct Surveys: A Step by Step Guide. Newbury Park, Calif.: Sage, 1985.
• Sudman, S., and Bradburn, N.M. Asking Questions: A Practical Guide to Questionnaire Design. San Francisco: Jossey-Bass, 1982.

• Brinberg, S., and McGrath, J.E. Validity and the research process. Newbury Park, Calif.: Sage, 1985.
• Carmines, E.G., and Zeller, R.A. Reliability and validity assessment. Newbury Park, Calif.: Sage, 1979.

• Bruning, J.L., and Kintz, B.L. Computational handbook of statistics. 3rd ed. Glenview, IL: Scott, Foresman, 1987.
• Conover, W.J. Practical nonparametric statistics. New York: Wiley, 1971.
• Schutte, J.G. Everything you always wanted to know about elementary statistics (but were afraid to ask). Englewood Cliffs, NJ: Prentice-Hall, 1977.