Multi-country grants face multiple challenges but there seems to be a consensus in what they should address

2. ANALYSIS
3 Nov 2021
The Global Fund needs to ensure that any future MCGs are more efficient and cost-effective in terms of technical, financial, and human resources and that they have measurable and demonstrable results

In addition to core allocations, the Global Fund channels funding through catalytic investments, including Multi-Country Grants (MCGs), to support activities that may not be readily addressed by individual country grants. These grants intend to add value by supporting activities to produce outcomes that go beyond what could be achieved by a single country grant, including tackling regional bottlenecks and addressing cross-border issues.

The Technical Reference Evaluation Group (TERG) commissioned an independent Thematic Evaluation of Multi-Country Catalytic Investment Grants to critically assess how the MCGs have been functioning and to what extent they have achieved their intended objectives. This thematic evaluation focused on the MCGs for the 2017-2019 allocation period, as well as the preparation of MCGs for the 2020-2022 period that have been exclusively supported by catalytic funding. The overall purpose of the evaluation was to examine the additional value added by MCGs that would not have been accomplished by country grants alone, including, but not limited to, whether they are achieving their catalytic effects. In agreement with the TERG, both in the request for proposals (RFP) and in subsequent discussions during the inception phase, the evaluation was designed to be strategic in its focus, to provide guidance to the Board and Strategy Committee (SC) on MCGs under the 2023-2025 allocation period.

The review covered eight regional grants as listed below:

Table 1. List of MCGs evaluated under the thematic evaluation

The main findings

Relevance of the topics addressed: the selection of MCG priorities has become progressively more focused and strategic from NFM1 to NFM2 and NFM3

Interviews with a range of stakeholders from across the Global Fund Board, SC, Grant Approval Committee (GAC), Secretariat and Technical Review Panel (TRP), and with technical partners, revealed the lack of shared understanding of the term ‘catalytic’. This constrained the effectiveness of the prioritization process for all catalytic investments including MCGs and resulted in the selection of MCGs that individually had a clear rationale and strong justification but whose catalytic potential was unclear. In the 2020-2022 allocation period, criteria for prioritization were developed based on replenishment levels.

Table 2. Catalytic priorities

Interviews with key informants at the regional level and in countries where MCGs were implemented reveal variable levels of country engagement in the selection process, with some, but not all, noting that regional and country stakeholders were consulted by Global Fund staff to ensure alignment with regional and national strategies and plans.

The evaluation reveals that MCGs are generally treated like country grants. Neither the TRP nor the GAC includes a specific review of the landscape analysis across all the participating countries. In particular, the GAC generally only receives inputs from the Fund Portfolio Manager (FPM) in the country where the Principal Recipient (PR) is based, not the individual country FPMs, so it may have a limited overview of the broader scope of the grant. Furthermore, the catalytic intent of the MCGs for the GAC is context-specific. TRP members noted that MCGs are more difficult to review because of the complexity of the implementation arrangements, governance structures, and technical approaches.

MCGs’ efficiency and value for money

The evaluation notes that performance indicators have evolved significantly from NFM1 to NFM2. Whereas NFM1 indicators were largely focused on outcomes and impact, NFM2 indicators were based on normative standards – Work Plan Tracking Measures (WPTM), inputs, and processes. Nevertheless, challenges related to indicators persist, such as:

  • The lack of agreement across countries on standardized indicators, when each country has historically had its own indicators and priorities;
  • Too short an implementation period and corresponding activities being implemented may not match with expected indicator levels (outputs versus outcomes versus impact), results, and targets;
  • Achievement of WPTMs are often dependent upon building the capacity of civil society organizations (CSOs), and the time needed to achieve this often exceeds the timeframe for MCG implementation;
  • Difficulties in comparing or aggregating country-level indicators, especially when data quality is questionable; and
  • Too few indicators address the level and quality of implementation, and these require significant dedicated time to identify implementation gaps.

Moreover, additional constraints were experienced in collecting and analyzing quality programmatic data at regional and country levels because of the following reasons:

  • There is too little country-level ownership of MCG performance measures, and hence little incentive for national programs to collect relevant data;
  • Some difficulties were highlighted in obtaining reliable size estimates and location data of hard-to-reach populations (e.g., key and vulnerable populations [KVPs]) that impedes the establishment of robust baselines, targets, and grant achievements;
  • Regional measures have either been delinked from country-level measures or there are no regional measures at all; data systems do not always consider differentiated approaches across countries to data collection in order to meaningfully aggregate the data; and
  • Administrative and legal requirements for sharing surveillance and information data are often cumbersome and restrictive; setting up regional databases is generally complex and time-consuming, due to constraints in alignment between different country data sets, and political barriers to the sharing of data across borders, especially for sensitive populations such as KVPs.

 

Table 3. Grant performance ratings for eight evaluated MCGs

The evaluation further notes that none of the NFM2 MCGs had a theory of change (TOC) or even robust results chains. Only two NFM3 MCGs were required to develop a TOC (MCG EECA APH and MCG SEA AFAO) and this was a new requirement in the RFP selection process. It was also noted that a robust TOC would describe the social and political environments in which the MCG must operate, together with the associated risks that may affect MCG outcomes. Finally, a TOC could identify what an MCG should achieve that would not be possible through a country grant alone.

The evaluation notes that MCGs have not been set up to measure or manage for value for money (VfM), and there is insufficient evidence to make an informed generalized assessment of VfM. Nonetheless, the analysis suggests that none of the MCGs have offered excellent VfM, three have offered good VfM, three have offered ‘adequate’ VfM, and two have offered ‘poor’ VfM. While MCGs have been targeted towards issues that country grants could not be expected to fully address, VfM has often been compromised by high Secretariat transaction costs and program management costs, poor grant design and issues in implementation, and poor performance against programmatic indicators.

Challenges in the implementation of the MCGs

Because of many MCGs’ complexity of implementation and governance arrangements, some associated risks were observed. Some of these include limited engagement and ownership at the country level of MCG implementation and results, weak/ inconsistent reporting by countries to regional databases, social and political constraints to sharing data across countries, inappropriate performance management frameworks, constrained MCG implementation periods resulting from delays in start-up, etc. The evaluation reaffirms the need to identify these risks during the design and review processes, and to ensure that risk mitigation measures are adequate and realistic.

Some key informants at global and country levels also noted that some TRP members, while possessing significant technical expertise, have limited experience in grant implementation and especially in MCG implementation. This may result in critical MCG implementation (and governance) risks not being identified or being under-prioritized during the review process.

Table 4. MCGs, their results, and linkages to the Global Fund’s Strategic Objectives

As noted in the 2016 TERG Thematic Review of the Global Fund’s Regional/Multi-country Grants, the linkages between MCGs and corresponding country grants were uneven. This finding has not changed in the five years since that review. Some in-country stakeholders knew little about MCG implementation in their country, while others, specifically engaged with the MCG, were not well informed about the country grant. The engagement of country grant stakeholders with the MCGs is visible during the early phases of the project cycle, as in-country stakeholders seek to ensure alignment with national and regional strategic plans and the country grants, as well as securing Country Coordinating Mechanisms’ (CCMs) endorsement of the MCG funding request. It is during implementation when broad knowledge of the MCGs and their results decline. This pattern of engagement is mirrored within the Global Fund Secretariat, where interactions between regional managers, FPMs for the MCGs, FPMs for the corresponding country grants, and Country Team (CT) members, remain dependent on personal working relationships and how much each FPM and/or CT member decides to engage with his/her counterparts.

As noted in the OIG 2019 Audit Report for Global Fund Multi-Country Grants, “most grants are overseen by different Global Fund Secretariat teams, meaning that the country has no single focal point at the Secretariat”. While some efforts have been made to strengthen coordination across MCGs, several stakeholders reported that these multiple interactions continue to cause coordination difficulties and frustrations at the country level.

How do MCGs effectively tackle regional bottlenecks and address cross-border and national issues?

The evaluation identified some minor positive unintended effects. MCG RMEI was noted for its promotion of regional achievements, the shared goal of eliminating malaria, and to a certain extent a sense of peer pressure, in that countries were not only tracking their progress but the progress of neighbouring countries. It was well-received within the region when El Salvador was certified by the World Health Organization (WHO) as malaria-free. A stakeholder from a neighbouring country noted, “This promotes a sense of participation and raises morale.” This was also true for MCG NTP/SRL where the “regionality” aspects fostered a healthy competition to achieve accreditation for national reference laboratories among the 23 participating countries.

Figure.1 Ability of MCGs to interact at the regional and country levels (N=43)

Many Regional Coordination Mechanisms (RCMs) experience challenges in their operationalization. In-country stakeholders reported critical challenges such as the coordination among civil society partners, the lack of recognition of the RCM by the PR or at national levels, and the limited interaction with CCMs and the country programs. Indeed, for some regional bodies, a conscious decision has been made to limit interaction with CCMs and high-level Ministry of Health counterparts to avoid administrative bottlenecks, in order that partners can work more strategically and efficiently to reach the goals of the MCGs

For the MCGs reviewed by the evaluation, regional platforms that could be considered successful in achieving the four core responsibilities of a CCM/RCM are the exception rather than the norm. Discussions with the Global Fund CCM Hub reveal that RCMs are now included in the continuation phase of the CCM Evolution Initiative, although they are presently not held to the same functionality standards as CCMs. Interviews with the Secretariat and at the regional level indicate that the Secretariat has not provided consistent guidance on how FPMs/CTs can best engage with and support RCMs. The diversity of RCMs suggests that a standardized approach to RCM support may not be effective. The CCM Hub notes that it has provided support to a limited number of RCMs, although this has been on an ad-hoc rather than a systematic basis. Interviews with regional and country-level stakeholders reveal that, where RCMs received support from the Global Fund Secretariat and the CCM Hub, in particular, this engagement has helped to strengthen their governance function. The Secretariat now requires RCMs to fulfill all six CCM eligibility requirements, including the establishment of an oversight body, in order to receive support from the Global Fund.

MCG partnership arrangements

As noted above, multiple key informants at global, regional, and country levels cited well-functioning partnerships as key to the success of the MCGs. However, some stakeholders expressed concerns about aspects of MCG partnership arrangements. These concerns included the quality and usefulness of technical assistance provided by some of the partners, in that due diligence was not always sufficiently undertaken in assessing the capacity of regional networks and partners, and the missed opportunities to build regional technical support.

Based on the eight MCGs examined, there are indications that some thematic areas and, perhaps, regions, lend themselves more to regional and sustainable capacity building than others. For example, MCG NTP/SRL is developing a TB Supranational Reference Laboratory Network for West and Central Africa and is therefore specifically designed to build regional capacity.

Lessons learned from the evaluation processes

These lessons learned are outlined in more detail below:

Risk identification & mitigation: because of the complexity of implementation and governance arrangements of many MCGs, there are associated multifaceted, diverse, and sometimes subtle risks. Some of these risks, outlined earlier in the findings, should be identified during the design and review processes to ensure that risk mitigation measures are adequate and realistic. Major risks should be highlighted in a robust TOC, which should be incorporated in the MCG design.

Communication and approval protocols associated with complex implementation arrangements (Regional PR-Country PRs/Co-PRs, SRs-SSRs) are constraining SSRs’ effective MCG implementation, including long delays between seeking and receiving approvals for program/budget modifications. Continued weaknesses of most RCMs/ROs, including limited oversight function, are constraining their governance functions, which could benefit from more proactive support from the CCM Hub.

Performance Measurement: A critical improvement would be for each MCG to have a robust TOC embedded in the design, with outcome, output, and intermediate indicators clearly identified before implementation. Each MCG is unique, so each should have an individualized Performance Framework and specific indicator set. WPTMs are generally developed top-down by the Secretariat and given to in-country counterparts who sometimes feel little ownership or accountability for them.

Duration of the grants: Multiple stakeholders at global, regional, and country levels highlighted the need to review the three-year funding cycle for MCGs, especially given inherent delays in start-up. While acknowledging the constraints imposed by the Global Fund replenishment cycle, stakeholders suggested approaches adopted by some other bi-lateral and multi-lateral funding agencies, where a five-year planning cycle is envisaged, with a review mid-cycle and course correction/ discontinuation if needed. A related issue concerns the synchronization of implementation timeframes for MCGs and country grants.

Recommendations

Resource mobilization, both domestically and accessing other donor funding, in order to address the limited three-year project cycle, the mobilization of additional resources was mentioned by many key informants as critical.

An enhanced partnership approach with new actors: Key informants noted the need for the Global Fund to move away from its “traditional” partnerships and explore new partnering opportunities. The private sector, for example, is seen as significantly underutilized.

Fund smaller grants that could be managed by other partners: MCGs, in general, have high transaction costs for the Global Fund Secretariat, especially when they are for relatively small amounts. Other partner organizations may be better placed to access regional partners and beneficiaries and could manage the grants directly on behalf of the Global Fund.

Supporting the development of future leaders to drive innovation: Interventions mentioned included twinning programs, targeted digital technical assistance to regional leaders and gatekeepers, and the continued development of regional networks.

A continued focus on cross-border issues: While this is currently applicable to mainly malaria and TB MCGs, with increased migration and potential pandemics (such as COVID-19), due to varying factors, it may need to be expanded to all diseases and resilient and sustainable systems for health.

According to the evaluation, there is a general desire to continue addressing issues which cannot be done via country allocations. Specifically for community and KVP rights, gender-based violence and violence against transgendered individuals, stigma and discrimination, advocacy, regional CSO strengthening, and region-wide policy changes. As noted by multiple stakeholders, the Global Fund needs to ensure that any future MCGs are more efficient and cost-effective in terms of technical, financial, and human resources and that they have measurable and demonstrable results.


Share |

Leave a comment

Leave a comment