Позитивные изменения, Том 3 №1, 2023. Positive changes. Volume 3, Issue 1 (2023)

EXTERNAL EVALUATION OF THE FOUNDATION’S GRANT COMPETITION

The contest is improved every year, so as to make it as objective, transparent, understandable and interesting for the participants as possible.

In September 2018, the Foundation’s team decided that an external evaluation was needed, which was conducted a year later. Two independent experts were engaged in the evaluation. The evaluation report was received in November 2019. It took 6 months to generate the scope of work, set up the questions for the evaluation and select the specialists. This period was the most productive for the Foundation’s staff in terms of obtaining information about its activities, analyzing the data accumulated over the years, determining the priorities, structure and logic of the activities, strategic development objectives, etc.

The decision to conduct the evaluation was not originally related to the grant competition. The choice to evaluate the competition rather than the programs/projects being implemented, approaches to management, monitoring and evaluation of results was made during the development of the scope of work, after communication with prospective evaluators and a series of internal discussions, including a conversation with the donor’s representatives.

The decision was made because the competition allowed the Foundation to address multiple problems. For example:

1) evaluate the needs and social problems faced by different target groups;

2) identify the readiness/maturity of various organizations, including in the non-profit sector, to address social problems and to develop social projects;

3) analyze these organizations’ readiness to establish partnership and interdepartmental interaction;

4) build mechanisms for interdepartmental interaction and partnership;

5) affect the positioning of the company and its CSR fund.

Thus, the competition has proven to be an important procedure affecting programs, the Foundation itself, inter-departmental collaboration, and the progress towards the strategic goals of The Road Home Program and the Foundation.

The following questions were asked during the evaluation: To what extent does the grant competition contribute to the program effectiveness? To what extent do the competition regulations, the format of the applications and the expert review system of applications allow you to evaluate the effectiveness of the project and the adequacy of the declared costs? What changes should be made to the grant competition regulations, application, and the project evaluation system?

Trying to build a tentative ranking of the usefulness of various stages of assessment, that is, the saturation with practical information for development, we arrive at the following:

1) Preparing the scope of work, choosing evaluators.

2) Evaluation process; communication with evaluators; feedback from evaluation participants.

3) Discussion, implementation of results, recommendations.

4) Evaluation report.

As a result of the evaluation, the Foundation team received a report with a list of suggestions and recommendations for further development of the grant competition, and the evaluation process created an image of the changes that need to be implemented.

Expert quotes from the evaluation report:

• “The mechanism of the competition is clear and well-established. The open competition format is convenient and should be retained as a form of selecting projects for funding. The calendar schedule of the competitions is also convenient.”

• “According to the interviewees’ opinions, the influence of the Foundation’s activities on positive social changes in the beneficiaries’ situation and on solving the city’s social problems (within the priority areas of operation) is estimated at 40 %. This score is very high.

• “A supported project, after several years of successful work, can get the status of the Foundation’s “service”. The methodology has been extensively elaborated, algorithms have been developed, and strong interdepartmental interaction has been established.”

TRANSFORMATION OF ACTIVITIES

The 2019–2020 evaluation resulted in a major transformation of all competitive procedures. The following key changes took place:

1. The competition is now part of the Foundation, not of its separate Road Home program.

The Foundation is implementing two special-purpose programs, but its activities are wider than the two Programs. In light of this, a separate target group was added: (child welfare specialists), and the grant competition has been expanded to accept projects aimed at crisis prevention work (media projects, public opinion-forming projects, projects to develop safe behavior skills).

2. The geography was also extended, first to the neighboring districts of the Vologda region, whose residents often commute to work in Cherepovets and whose children attend the educational institutions in the city. Second, a large regional center (Vologda) was also covered by the project. The third city added to the competition was Veliky Ustyug in Vologda Region, where Sveza company (an active supporter of The Road Home Program) has its office. Finally, Yakovlevo settlement in Belgorod Region.

3. Purposeful collection of feedback from applicants and experts has started, in order to make the necessary changes to the competitive procedures promptly. After the competition, we hold consultations and meetings with those applicants whose projects did not receive funding. Unsuccessful applications may be reworked together with the specialists from the Foundation’s Resource and Methodology Center and submitted to grant competitions by other grantmaking organizations.

4. The principle of variability of contest mechanisms was introduced: projects can be selected through an open competition, initiated by Foundation employees after analyzing the problems and the current situation in their territories, or the Foundation’s specialists can participate in partners’ projects.

5. The competition has received its own website: https://socialprojects.dorogakdomu.ru/. Applications are accepted only through the applicant’s personal account; experts also have to log in to review the applications.

6. The process of building the evidence base of the project starts during the preparation of the grant application (new sections have been added to the document).

7. The Foundation has defined the support functions for the competition (administrators, thematic consultants, programmers).

8. A three-tiered application review system has been tested and approved.

9. Independent experts are involved in the evaluation, which eliminates the conflict of interest for applications submitted by experts themselves.

10. A new body, the Competition Council was introduced.

Thus, as we can see, the evaluation has significantly increased the maturity of the competition, which also improved other processes of the Foundation: inter-departmental interaction mechanisms, positioning of the Foundation, planning of activities, project management (see Table 1).

Table 1. Immediate results of the Foundation’s grant competition after implementing the independent evaluation results

INTERNAL EVALUATION AND SELF-EVALUATION OF THE COMPETITION

These types of evaluations are conducted annually, following the principle of triangulation by data source. That is, information about the competition is collected from at least three data sources, to enable independent validation of the results and examine the evidence of social outcome from different perspectives.

1. Every year since 2008, discussions are held before and after the competition with the team of organizers. The analysis focuses on the following issues: directions and target groups of the competition, geography, and the contribution of grantees’ projects to the achievement of the Foundation’s strategic goals.

2. Since 2020, feedback has been collected annually from applicants and members of the competition committee. The data collection method is a blanket survey. An online questionnaire is used. In 2021, 66 % of the applicants and 81 % of the members of the competition committee participated in the survey. Here are some insights from the 2021 report on the feedback from the applicants and experts of the competition:

• “For the convenience of applicants and experts in 2020, reference materials were developed: a manual for the experts and a manual for the applicants. Almost all applicants and experts, with the exception of 7 % and 5 %, respectively, took advantage of the manual. All of them noted that the reference material was helpful when working with the competition site.”

• “60 % of the applicants received advice from the Foundation specialists by phone or email while preparing their applications. Communication with specialists was highly appreciated by the applicants and experts of the competition. Commenting on their evaluation, experts and applicants noted that consultations were received promptly, and the specialists were able to answer all questions of interest. Applicants who did not seek counseling say they were able to complete the application on their own, thanks to detailed instructions in the applicant’s manual. The experts thanked the entire competition team for their quick feedback, attentive and friendly attitude.”

3. Beginning in 2019, the professional community (evaluators, specialists of grantmaking organizations, and attendees of the evaluation conferences conducted by the Association of Program and Policy Evaluators) are involved in the discussions of competitive procedures.

FEATURES OF THE GRANT COMPETITION

1. The Road Home Charitable Foundation regularly conducts sociological and evaluation research on relevant issues. Research results are published on the organization’s official website and discussed in professional circles[56 - The Road Home Program. (2023). The Road Home Charitable Foundation’s Resource and Methodology Center: we study problems, you offer solutions. Retrieved from: https://dorogakdomu.ru/resursno-metodicheskij-czentr-bf-doroga-k-domu-my-izuchaem-problemy-vy-predlagaete-resheniya/. (accessed: 06.03.2023).]. This ensures that potential applicants do not have to conduct a pre-project evaluation on their own, or go out looking for a problem or a vulnerable social group. The applicants can freely obtain this data from the research results.3

Information about the competition is collected from at least three data sources, to enable independent validation of the results and examine the evidence of social outcome from different perspectives.

2. Support to implementation of winning projects. Throughout the implementation period, the grantees are provided various kinds of support: information support (promotion, raising awareness, boosting the media coverage of the project); volunteer assistance; and methodological assistance.

3. Introducing an evidence-based approach at the application stage[57 - YouTube.com. (2021). Alena Bogomolova’s workshop “Approaches to Monitoring and Evaluating Child Welfare Activities.” Retrieved from: https://youtu.be/vSc2K5iymWY. (accessed: 06.03.2023).].

4. The competition has been extended to Cherepovets. Cherepovets is a medium-sized industrial city in Russia, which for several decades had the status of a single-industry city, with the city-forming enterprise of Severstal. The Foundation was the first to hold a grant competition in the city. The grantees also gained access to training, specialized methodological materials, expert support, a professional community, and the development and testing of new technologies for working with children and families. One of the conditions for implementing projects is inter-departmental collaboration, partnership with specialists from the social sphere, healthcare, education, law enforcement, higher education, local authorities, businesses, and media. As a result, the city developed a pool of experts and partners who specializing on working with troubled children, know how to use the latest technologies of social work, share the Foundation’s values, apply project-based approach and implement monitoring and evaluation tools in their work. Due to Cherepovets being a relatively small city with limited human resources, this pool of specialists and partners took part in the Foundation’s activities in different roles.