Program Evaluation Part 3 – My Favorite Evaluation Theories

happy sad icons

Introduction

Evaluations are part of our everyday lives. And yet, so few know and implement program and project evaluations in a logical and meaningful way. This is Part 3 of four-part blog series that aims to expand our collective understanding of the definitions, kinds, and implementations of evaluation and evaluation research. Part 3 focuses on reflections and critiques of some evaluation theories that spoke to me. Click here for Part 1. Click here for Part 2.

Reflections on and Critiques of Some of My Favorite Evaluation Theories

Responsive evaluation (Stake, 2013) uses informal evaluation approaches, including "getting personally acquainted with the evaluand, observation of activities, interviewing people in different ways familiar with the evaluand, [and] searching documents that reveal what happened in the past" (p. 191). Multiple measures are taken with the goal of something more akin to ethnography, which describes the evaluation thoroughly and narratively, rather than with the goal of generalizability. I like responsive evaluation because I think all evaluation should be responsive in some way. Every evaluation should thoroughly describe the context and use narrative qualitative methodology. However, I do think responsive evaluation might be a bit too informal, and, while I understand the need to be responsive to clients and stakeholders, I think lessening the importance of comparative standards within this framework undermines the value ascribed to the evaluation. 

David Fetterman, the "father" of empowerment evaluation, describes empowerment evaluation as "the use of evaluation concepts, techniques, and findings to foster improvement and self-determination" (Fetterman, 2013, p. 305) and a framework that empowers the evaluation and its stakeholders to "plan, implement, and evaluate their own programs" (Wandersman et. al, 2005, p.7 as found in Fetterman, 2013, p. 305). Empowerment evaluation, while based on theories, guided by ten principles, and implemented using key concepts and steps, is essentially about people, particularly the stakeholders and their ability to transform their own programs. The point of this kind of evaluation is to "cultivate a culture of evidence by asking people why they believe what they believe" (Fetterman, 2013, p. 311) and to initiate cycles of individual and group reflection so as to improve the program and the process.

While I found Fetterman's chapter a bit confusing – there is too much information in too few pages – I enjoyed interacting with him during a class presentation. There is something really interesting in providing the tools to empower a group of program stakeholders to conduct their own evaluative processes. I think a better term than culture of evidence is culture of evaluation. A culture of evaluation is a much needed framework, and my main critiques - including objectivity, bias, and transferability - are ones Fetterman addresses, although not to my satisfaction. He never addresses how stakeholders:

  1. pick the best methodology for their evaluation (or how they would even know the methodologies available);
  2. systematically try to eliminate bias from their process and outcomes; and
  3. increase the transferability and credibility of the evaluation plan and implementation.

Within the chapter, there is very little information on how to document the process and findings of the evaluation. Overall, there were several blanks stakeholders would need to fill in. But the sheer importance of the idea of a culture of evaluation from this empowerment evaluation framework may make these other concerns less important overall. 

Hallie Preskill defines appreciative inquiry as "a process that builds on past successes (and peak experiences) in an effort to design and implement future actions" (Christie, 2006, p. 466). I think focusing on the positive psychology aspects of an evaluation is a critical theme for improving the usefulness of the evaluation as well as increasing the buy-in from stakeholders. The positive psychology theory underlying appreciative inquiry includes using positivity (a constructionist method), a poetic or narrative based data (collection and analysis) focus, inquiry to drive change (simultaneity), and imagination to drive action (anticipatory) (Tocino-Smith, 2020). The challenge in using appreciative inquiry seems to be evaluating the growth edges of the evaluation and simultaneously the strengths while focusing solely on the strengths as much as possible. I think the psychological benefits of this mindset (to the evaluator, the evaluator, and the stakeholders) far outweigh any critiques of objectivity or Pollyanna lenses.

Culturally responsive evaluation (CRE), a term Stafford Hood first used, "recognizes that culturally defined values and beliefs lie at the heart of any evaluative effort" (Hood, Hopson, & Kirkhart, 2015, p. 283). Designing, implementing, and communicating an evaluation to a larger audience requires prioritizing "the needs and cultural parameters of those who are being served" (Hood et. al, 2015, p. 308). CRE pays special attention to those populations that have been marginalized. The main points of emphasis in CRE include focusing on lived experiences (of the evaluator and the evaluated), paying attention to power differentials, and embracing social justice.

While I do not like the term "culturally responsive evaluation" as it seems purposefully nebulous, I do like the focus in this kind of evaluation on deep understanding of the cultural context of the evaluand and the evaluated first. I also appreciate the emphasis on being particularly sensitive to marginalized groups while trying to evaluate and empower them. This idea fits into my studies on critical pedagogy and critical race theory which have now become integral parts of my teaching and research philosophies. It is nice to recognize that culturally responsive evaluation already exists and that it integrates these concepts.

References:

Christie, C. A. (2006). Appreciative inquiry as a method for evaluation: An interview with Hallie Preskill. American Journal of Evaluation, 27(4), 466-474. https://journals.sagepub.com/doi/10.1177/1098214006294402

Fetterman, D. M. (2013). Empowerment evaluation: Learning to think like an evaluator. In M.C. Alkin (Ed.), Evaluation roots: A wider perspective of theorists' views and influences (2nd ed., pp. 304-322). Sage Publications. https://doi.org/10.1177/1098214013479152

Hood, S., Hopson, R. K., and Kirkhart, K. E. (2015). Culturally responsive evaluation: Theory, practice, and future implications. In K. E. Newcomer (Ed.), Handbook of Practical Program Evaluation (pp. 281-317). John Wiley & Sons Incorporated. https://onlinelibrary.wiley.com/doi/book/10.1002/9781119171386

Stake, R. (2013). Responsive evaluation IV. In M.C. Alkin (Ed.), Evaluation roots: A wider perspective of theorists' views and influences (2nd ed., pp. 189-197). Sage Publications. https://doi.org/10.1177/1098214013479152

Tocino-Smith, J. (30 September 2020). How to apply appreciative inquiry: A visual guide [Online Article]. Positive Psychology. https://positivepsychology.com/appreciative-inquiry-process/

Community: