Позитивные изменения, Том 3 №1, 2023. Positive changes. Volume 3, Issue 1 (2023)

Evaluation Practice, 7(1), 63–64. Retrieved from: https://journals.sagepub.com/doi/ abs/10.1177/109821408600700105.

4. Kuzmin, A. (2009). Use of evaluation training in evaluation capacity building. In M. Segone (Ed.), From policies to results: Developing capacities for country monitoring and evaluation systems. New York: UNICEF.

5. Kuzmin, A., Karimov, A., Borovykh, A., Abdykadyrova, A., Efendiev, D., Greshnova, E…. Balakirev, V. (2007). Program Evaluation Development in the Newly Independent States. Journal of MultiDisciplinary Evaluation, 4(7), 84–91. Retrieved from: https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/14/29.

6. Love, A. J. (1983). The organizational context and the development of internal evaluation. New Directions for Program Evaluation, 20, 5–22.

7. Love, A. J. (1991). Internal Evaluation: Building Organizations from Within: SAGE Publications.

8. McClintock, C. (1983). Internal Evaluation: The New Challenge. American Journal of Evaluation, 4, 61–62.

9. Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks: Sage Publications.

10. Rogers, A., McCoy, A., & Kelly, L. M. (2019). Evaluation Literacy: Perspectives of Internal Evaluators in Non-Government Organizations Canadian Journal of Program Evaluation / La Revue canadienne d’еvaluation de programme, 34(1), 1–20. Retrieved from: https://journalhosting.ucalgary.ca/index.php/cjpe/article/view/42190.

11. Volkov, B. B. (2011). Internal evaluation a quarter-century later: A conversation with Arnold J. Love. New Directions for Evaluation (132), 5–12.

12. Volkov, B. B., & Baron, M. E. (2011). Issues in internal evaluation: Implications for practice, training, and research. In B. B. Volkov & M. E. Baron (Eds.), Internal evaluation in the 21st century. New Directions for Evaluation. (Vol. 132, pp. 101–111).

13. Альянс PROОЦЕНКУ. (2023). Принципы самооценивания программ и проектов в НКО. Режим доступа: https://www.processconsulting.ru/attach_files/menu_21_12_22_19.pdf. PROOCENKU Alliance (2023). Principles of self-evaluation of NGO programs and projects. Retrieved from: https://www.processconsulting.ru/attach_files/menu_21_12_22_19.pdf.

14. АСОПП. (2017). Принципы оценки программ и политик. (05.02.2023). Режим доступа: https://eval.ru/attach_files/file_menu_78.pdf. ASPPE. (2017). Principles of program and policy evaluation. (05.02.2023). Retrieved from: https://eval.ru/attach_files/file_menu_78.pdf.

15. Кузьмин, А. И. (2016). Будущее оценки. Режим доступа: https://evaluationconsulting.blogspot.com/2016/06/blog-post_14.html. Kuzmin, A. I. (2016). The Future of Evaluation. Retrieved from: https://evaluationconsulting.blogspot.com/2016/06/blog-post_14.html.

16. Кузьмин, А. И. (2020). История развития мониторинга и оценки в Сети СЦПОИ. Начало. Режим доступа: https://evaluationconsulting.blogspot.com/2020/04/blog-post.html. Kuzmin, A. I. (2020). History of the development of monitoring and evaluation in the SCISC Network. Beginning. Retrieved from: https://evaluationconsulting.blogspot.com/2020/04/blog-post.html.

“Do it yourself:” How NGOs Evaluate Projects Using Their Own Resources

Many experts believe that self-evaluation is one of the key trends in the development of evaluation in general. Self-evaluation is similar to external evaluation, but it has its peculiarities. The authors of this article discuss the importance of evaluation principles, and provide recommendations for NGO self-evaluation situations.

Alexey Kuzmin

PhD, Director General, Process Consulting Company, Council Member, International Evaluation Academy

THE STORY OF ONE PREDICTION

In 2012, American evaluator John Gargani, who later became president of the American Evaluation Association, posted 10 predictions in his blog about the future of program and project evaluation. The first point was worded as follows: “Most evaluations will be internal.” (Gargani, 2012).

What is it about?

To begin with, program (project) evaluation is “the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/ or inform decisions about future programming.” (Patton, 2008).

Suppose an organization is implementing a project. The project can be evaluated by specialists external to the organization. This is external evaluation. Alternatively, the project can be evaluated by the specialists within the organization itself. This case is referred to as internal evaluation. Sometimes self-evaluation is singled out from internal evaluation, when the project is evaluated by the very same people who implement it.

Active development of the evaluation of socially oriented projects and programs in the United States started in the 1960s. The requirement to conduct an evaluation was part of the “top-down” approach, utilized first in the educational institutions, and then in other areas. This was an innovation at the time, and the evaluation was solely the domain of external consultants with the appropriate competencies.

It soon became clear that organizations should develop their own evaluation capabilities. The relevant practical experience was accumulated over the course of 10 to 15 years. The development of internal evaluation practices was one of the central topics at the 1982 American Evaluation Association conference (McClintock, 1983).

Arnold Love, Canada (A. J. Love, 1983), noted that internal evaluation serves two basic functions: first, it provides relevant and timely information for management decision-making; and second, the internal evaluation process in itself has a positive effect on both decision-making and the organizational performance. Love suggested that “over the next few years we can expect the internal evaluation function to continue growing and changing” (A. J. Love, 1983). By the way, this prediction was published 30 years before the one made by John Gargani.

Around the same time, Ernest House (1986) observed that the practice of public organizations in the United States was shifting from external evaluations, conducted primarily by universities and private companies, to internal evaluations conducted by specially created units. At the same time, most of the publications available at the time referred specifically to external evaluation. That is why the publication of A. J. Love’s book, “Internal Evaluation: Building Organizations from Within” (A. J. Love, 1991) became a remarkable event. The author of this book remains one of the most quoted authors to this day, when it comes to internal evaluation.

Table 1. Benefits and limitations of external and internal evaluation

20 years later, the “New Directions in Evaluation” series[28 - Publication of the American Evaluation Association.] publishes a collection of works titled “Internal Evaluation in the 21st Century.” Isn’t it somewhat old for a new direction?

In an interview with the book’s editor (Volkov, 2011), Arnold Love recalls that in the 1990s up to 60 percent of all evaluations in the United States were conducted internally. At the same time, internal evaluation was developing differently from country to country. In Canada, for example, the federal government decided back in late 1970s that evaluations of government programs should be primarily internal, while the Japan Evaluation Society reported in 2006 that 99 % of evaluations in their country were internal.

Even though internal evaluation is being actively implemented all over the world, the number of studies and publications on this topic in the second decade of the 21st century is still very limited (Volkov & Baron, 2011). However, the issues of developing internal evaluation in smaller organizations with limited resources are increasingly coming into spotlight (Baron, 2011; Rogers, McCoy, & Kelly, 2019).

Practice shows that internal evaluation can coexist happily with external evaluation, and most experts now agree that these are not mutually exclusive, but complementary approaches (see Table 1).

Now back to John Gargani’s prediction (Gargani, 2012). He suggested that internal evaluation in the U. S. would largely replace external evaluation within 10 years. Thus, most program and project evaluations in the United States would be internal by 2022. Apparently, this has not happened yet, but we can definitely state that internal evaluation is developing in both government and non-government organizations in the United States, Canada, Japan, and many other countries. Russia is no exception.

INTERNAL EVALUATION IN RUSSIA

Without going too much into historical detail, we should note that in Russia everything started with external evaluation, which was “imported” to us in the early 1990s along with the international assistance programs by numerous foreign organizations (Kuzmin et al., 2007). Gradually, local evaluators started appearing as well. At the same time, evaluation developed in two directions: the market for external evaluation services was being established alongside the development of internal evaluation function.

The Russian NGOs needed to develop their own evaluation capacity for at least two reasons: 1) donor organizations’ requirements, and 2) the need to build management control systems due to increasingly complex and expanding NGO activities (Kuzmin, 2020). However, the majority of Russian NGOs failed to pay attention to developing evaluation competencies for quite some time.

The situation has only started in recent years:

• there has been a marked increase in the demand for training for NGO managers and staff in the field of evaluation;

• NGOs have learned to collect and process data much more efficiently;

• a number of NGOs have acquired successful internal evaluation experience;

• donor organizations increasingly require NGOs to submit analytical reports;

• at the annual conferences of the Association of Specialists in Program and Policy Evaluation (ASPPE)[29 - Association of Program and Policy Evaluators. (2023). Retrieved from: www.eval.ru. (accessed: 14.02.2023).], there has been a noticeable increase in the number of presentations on experiences in developing internal evaluation and its importance for high-quality organization management.

Six years ago, as I was observing these processes and comparing them to what was happening abroad, I suggested that by 2026 Russia would see “an increase in the evaluation capacity of various organizations, the development of internal evaluation and self-evaluation” (Kuzmin, 2016). Looks like this prediction is coming true. Another proof of this is the creation of the PROOCENKU Alliance[30 - The founders of the Alliance are Arkhangelsk Center of Social Technologies “Garant”, Process Consulting Company (Moscow), and the Siberian Civic Initiatives Support Center (SCISC, Novosibirsk).] in 2021 and the opening of an online discussion club of the same name, focused mainly on the development of internal evaluation and self-evaluation:

“The PROOCENKU Club is an ongoing online discussion platform for anyone interested in evaluation. The Club’s slogan goes as follows: “An interesting conversation in a good company.” The Club’s mission is to support a community of practitioners dedicated to integrating evaluation into the work of socially oriented organizations. Nothing is required to participate in the meetings, other than the desire. Education, level of training in evaluation, and practical experience are not important.”[31 - #PROOCENKU Alliance. (2022). Retrieved from: http://www.proocenku.club/. (accessed: 14.02.2023).]

PRINCIPLES OF SELF-EVALUATION OF NGO PROGRAMS AND PROJECTS

ASPPE has developed and published a number of important works on the professional development of professional evaluators. These include the Principles of Program and Policy Evaluation (ASPPE, 2017).

Professional principles are born out of practice and determine how members of the profession should behave in situations of difficult choices. Principles cannot be invented before real experience has been accumulated: it is a natural evolutionary process. When a profession reaches a certain level of maturity and a professional community (association) has been established, that community comes together to agree on the principles. Therefore, the principles of program and policy evaluation adopted by ASPPE are important not only as guidelines for action, but also as evidence of the emerging evaluation profession in Russia.

These principles are designed to take into account the fact that, generally, three parties can participate in the evaluation:

• The Customer is a representative of the organization that initiates the evaluation, orders the task to be implemented, and will be the main user of the evaluation results.

• An evaluator – a specialist hired by the Customer to perform the evaluation.