Advantages and disadvantages of evaluation checklists and how to use them to improve evaluation practice. Presented at USF Center for Research, Evaluation, Assessment, and Measurement.

1. The Evaluation Checklist Wes Martz, Ph.D.

2. A mnemonic device that consists of a list of activities, items, and criteria used to perform a certain task. 2

3. Guidance for the collection of relevant evidence used to determine the merit, worth, or significance of an evaluand. 3

4. Existing Evaluation Checklists Evaluation management Evaluation models Evaluation values and criteria Metaevaluation Evaluation capacity building Checklist creation 4 http://www.wmich.edu/evalctr/checklists

5. Checklist Organizer 5

6. Good Checklists 6

7. Criteria to Evaluate Checklists Applicability to full range of intended uses Clarity Comprehensiveness Concreteness Ease of use Fairness Parsimony Pertinence to the content area 7 Stufflebeam, D. L. (2000) Guidelines for developing evaluation checklists: The checklist development checklist. Available at http://www.wmich.edu/evalctr/checklists/guidelines_cdc.pdf

8. Checklist Advantages Consolidate vast knowledge in a parsimonious manner Improve task performance Reduce influence of halo and Rorschach effects Reduce resource use Improve memory recall Set out minimum necessary steps in a process 8 Scriven, M. (2005). Checklists. In S. Mathison (Ed.), Encyclopedia of evaluation (pp. 53-59). Thousand Oaks, CA: Sage.

9. Checklist Disadvantages Evaluation myopia Inappropriate use Fatigue resulting from overuse Unnecessary complexity decreases reliability Burdensome process delays completing evaluation 9 Martz, W. (in press). Validating an evaluation checklist using a mixed method design. Evaluation and Program Planning (2009). doi: 10.1016/j.evalprogplan.2009.10.005

10. Organizational Effectiveness Evaluation Checklist (OEC) Developing and Validating an Evaluation Checklist: A Case Example 10

11. OEC Overview Organizational evaluation process framework Iterative, explicit, weakly sequential Six steps, 29 checkpoints Criteria of merit checklist 12 universal criteria of merit 84 suggested measures 11

12. OEC Overview 12

13. Criteria to Evaluate the OEC Applicability to full range of intended uses Clarity Comprehensiveness Concreteness Ease of use Fairness Parsimony Pertinence to the content area 13

14. OEC Validation Process Phase 1: Expert panel review Critical feedback survey Written comments made on checklist Phase 2: Field test Single-case study Semi-structured interview 14

15. Expert Panel Overview Study participants Subject matter experts (organizational and evaluation theorists) Targeted users (professional evaluators, organizational consultants, managers) Review OEC for providing critical feedback Identify strengths and weaknesses Complete the critical feedback survey Write comments directly on the checklist 15

16. Expert Panel Data Analyses Critical feedback survey Descriptive statistics Parametric and nonparametric analysis of variance Written comments on checklist Hermeneutic interpretation Thematic analysis to cluster and impose meaning Triangulation across participants to corroborate or falsify the imposed categories 16

18. Representativeness

20. Observations from Field Test Structured format minimized “scope-creep” Identified several areas to clarify in OEC Reinforced need for multiple measures,transparency in standards Minimal disruption to the organization 19

21. Validity Study Assessment Strengths Relatively quick validation process Based on relevant evaluative criteria Features a real-world application Weaknesses Single-case field study Selection of the case study Selection of the expert panel members 20 Martz, W. (in press). Validating an evaluation checklist using a mixed method design. Evaluation and Program Planning (2009). doi: 10.1016/j.evalprogplan.2009.10.005

22. Lessons Learned Checklist development should address unique attributes of the evaluand Sampling frame is critical Checklist validation should be grounded in theory, practice, and use Mixed method approach provides increased confidence in validation conclusions All checklists are a “work-in-process” 21

23. Additional Checklist Resources Martz, W. (in press). Validating an evaluation checklist using a mixed method design. Evaluation and Program Planning (2009). Scriven, M. (2007). The logic and methodology of checklists. Available at http://www.wmich.edu/evalctr/checklists/papers/logic&methodology_dec07.pdf. Stufflebeam, D. L. (2000) Guidelines for developing evaluation checklists: The checklist development checklist. Available at http://www.wmich.edu/evalctr/checklists/guidelines_cdc.pdf. Stufflebeam, D. L. (2001). Evaluation checklists: Practical tools for guiding and judging evaluations. American Journal of Evaluation, 22, 71–79. 22 http://www.wmich.edu/evalctr/checklists

24. The Evaluation Checklist martz@EvaluativeOrganization.com @wesmartz