Strategic Initiatives align with Global Fund Strategic Objectives but only cover some of the challenges affecting the achievement of the Fund’s goals
Between April and July 2021, the Technical Evaluation Reference Group (TERG) commissioned an evaluation to assess the extent to which Strategic Initiatives (SIs) have achieved their intended objectives. The evaluation revealed that although SIs are aligned with Global Fund’s objectives and are relevant to programmatic needs, they only address some of the important challenges affecting the realization of the organization's goals. It is unclear why some of these key issues were selected over others. Despite being designed to accelerate progress in certain areas, the evaluation revealed they had failed to achieve this, although they had become more focused over their implementation period. Moreover, the SIs lacked an adequate coordination mechanism to dovetail their inputs with other catalytic funding and country allocations, thereby compromising their effectiveness due to missed opportunities, duplication and overlap. However, the evaluation noted that the Secretariat and partners improved engagement over time to coordinate and implement activities for most SIs, thereby nurturing greater transparency and accountability.
SIs are one of the three Global Fund’s mechanisms for catalytic investments, the others being Matching Funds and Multi-Country Grants, that support country allocations but cannot be funded through country grants due to their cross-cutting nature. For the 2017-2019 allocation period, the Global Fund invested $172 million in SIs, accounting for 21.5% of the $800 million for catalytic investments. The proportion of investment increased during the 2020-2022 allocation period as the Global Fund allocated $343 million to SIs, accounting for 38.5% of the $890 million allocated to catalytic investments.
The objectives and data sources for the evaluation
The TERG commissioned the evaluation to assess:
- The basis of the SIs’ selection, design, and prioritization, and their appropriateness for reaching their intended outcomes;
- The Sis’ implementation and monitoring arrangements; and
- The extent to which the SIs have achieved their catalytic effects.
The evaluation team obtained data from various sources, including desk reviews of relevant documents, interviews with key informants, and the development of case studies. For desk review, the team obtained information from reports, reviews and documents from the Global Fund Board, Technical Review Panel (TRP), Office of the Inspector General (OIG), and TERG. The evaluation interviewed a wide range of key informants drawn from:
- The Global Fund Board and Secretariat;
- Bilateral and multilateral agencies such as World Health Organization (WHO), the Joint United Nations Programme on HIV/AIDS (UNAIDS), and the United States Agency for International Development (USAID);
- private foundations such as Bill and Melinda Gates Foundation; and
- In-country key stakeholders such as Country Coordinating Mechanisms (CCMs), Principal Recipients (PRs), and civil society organizations (CSOs).
Due to the COVID-19 pandemic restrictions, the evaluation team conducted all interviews virtually. The team examined ten SIs in-depth, of which seven were continued from the 2017-2019 to the 2020-2022 allocation periods, while the remaining three were new SIs in the 2020-2022 allocation period, The team developed case studies on two countries for each SI.
The evaluation team structured its analysis according to three pillars:
- The selection and prioritization of SIs for both the 2017-2019 and 2020-2022 allocation periods, focusing on their relevance and coherence;
- Implementation arrangements for SIs for the 2020-2022 allocation period, focusing on their efficiency; and
- SIs’ results for the 2020-2022 allocation period, focusing on their effectiveness and value for money.
Key evaluation findings
The evaluation team highlighted several key findings, conclusions, and recommendations for TERG to consider.
Selection and prioritization of SIs
The evaluation reveals that the SIs selected for the 2017-2019 allocation were disjointed. It was unclear to various Global Fund organs, such as the Board and the Grant Approval Committee (GAC), how SIs would achieve their catalytic effect, and there were no metrics to measure their catalytic intentions. However, there was an improvement in the prioritization process for selecting SIs in the subsequent 2020-2022 allocation period. This improvement is attributed to lessons learned during the 2017-2019 allocation period, as all except two SIs were carried over to 2020-2022. Moreover, replenishment levels were part of the prioritization criteria used to decide which SIs should be continued. However, six new SIs were selected for the 2020-2022 allocation period. They had the same problem of lack of coherence in their strategic intents but focused on issues that are important to address country needs.
Regarding the SIs’ design, the evaluation assessed whether their catalytic intent was clearly spelled out in the 36th Board Paper and included a Theory of Change (TOC) showing how SIs’ activities are linked to desired outcomes. It found that the catalytic intent of seven of the 13 SIs in the 2017-2019 allocation period was not clearly articulated and only one SI had a complete TOC. In comparison, all SIs except one in the 2020-2022 allocation period have been designed such that their catalytic intent is adequately reflected, and most had some form of a TOC, albeit lacking some of the critical elements of a TOC such as clearly articulated risks and assumptions and pinpointing where the ‘catalysts’ are expected to occur.
The evaluation also revealed poor county-level engagement in SI design including regarding the timing of their implementation, which can inhibit their catalytic potential. However, some SIs’ interventions in the 2017-2019 allocation period supported the accelerated attainment of some outcomes that could not have been achieved using country allocations alone. For instance, the SI for catalyzing the market entry of new bed-nets has managed to expand the long-lasting insecticidal bed-nets (LLINs) market by including the next generation of LLINs that have an increased efficacy against pyrethroid-resistant mosquitoes. By leveraging additional funding from Unitaid, this SI has contributed to the increased availability of next-generation LLINs by providing volume guarantees to manufacturers and subsequently influencing a drop in the unit price for each net.
SI implementation arrangements
The evaluation showed that over time there had been a significant improvement in the SIs’ management and implementation arrangements, particularly during the 2020-2022 cycle. This improvement can be seen in the areas of SI coordination, contracting implementers, planning, implementation, and reporting. However, there were accountability concerns regarding the non-competitive selection process for technical partners.
Establishing good communication channels is key to tracking SI progress. During the 2017-2019 cycle, good communication channels were adopted by the Secretariat SI teams and between partners. For instance, the CCM Evolution SI conducted CCM Working Group Partner Meetings with members from several bilateral and multilateral agencies, providing a platform for sharing information and lessons, thereby developing a strong partnership approach. These robust communication channels will hopefully continue in the 2020-2022 cycle.
There were fears that the slow start-up of most SIs during the 2017-2019 cycle due to contracting delays, and the COVID-19 pandemic interruptions, would affect the completion of planned activities. For instance, COVID-19 interruptions affected technical assistance (TA) missions as training activities in most countries were canceled and the missions were forced to switch to virtual modalities. However, most of the planned activities for most of the reviewed SIs were completed by the end of 2020.
Sound performance monitoring indicators are essential for laying out the expected results and for transparency and accountability purposes. Most SIs for the 2017-2019 cycle lacked clear monitoring indicators, translating into vague results. This contributed to ambiguous partner accountability and transparency, particularly regarding who took responsibility for incomplete activities. During the 2020-2022 cycle, the situation changed as all reviewed SIs had clearer results frameworks with robust metrics aligned with reporting tools.
The evaluation revealed that there are concerns about the quality assurance process for TA. There has been little assessment or no systematic evaluation of the quality of TA deliverables for the 2017-2019 cycle. This is risky as it affects partnership and coordination.
The extent to which SIs met their objectives, were catalytic and contributed to the Global Fund’s Strategic Objectives (SOs)
Although there was a delay in launching some SIs for the 2017-2019 cycle due to various reasons noted above, in general, all SIs made progress towards their stated objectives. The pandemic stalled the implementation of SI components at the country level but less so for those which were centrally or regionally managed. However, there are certain objectives that the SIs cannot achieve within a three-year funding cycle and require a longer timeframe. For instance, WHO's definition of malaria elimination is that a country does not record any indigenous malaria cases for three consecutive years. Thus, countries with local malaria cases in the first year of implementing the Malaria Elimination SI required longer than three years to reach elimination status.
The evaluation adopted the definition of attaining catalytic effect as at least one of the four aspects of: (i) increasing funding or implementing more activities; (ii) improving the efficiency and effectiveness of previously implemented activities and/or make them more strategic; (iii) being unique, new, and innovative; and (iv) accelerating the pace of activities’ implementation. While some SIs are yet to demonstrate their catalytic effect, others such as the New Nets, CCM Evolution, Data, and TB Missing Cases, have done so.
As discussed above, the New Nets SI combined country net orders to increase the volume of orders thus facilitating a negotiated drop in the unit price of each net sooner than expected. The CCM Evolution SI introduced a new area of CCM linkages, improved some CCMs’ roles in developing co-financing commitment tracking tools, and accelerated the approval of revised CCM Guidelines and Requirements. The Data SI improved surveillance efficiency and applied new methods to increase data standardization and availability. The TB Missing Cases SI introduced previously underutilized technologies and new methods for TB case finding.
Although it is hard to quantify the extent to which the SIs contributed to the Global Fund’s SOs, all the reviewed SIs for the 2017-2019 cycle contributed to one or more SO. The TB Missing Cases and Malaria Elimination SIs contributed to maximizing the Global Fund’s impact against TB and malaria by reducing cases and mortality. The CCM Evolution, Data, Procurement and Supply Management, Service Delivery Innovations Labs, and Sustainability Transition and Efficiency SIs contributed to building resilient and sustainable systems for health. Moreover, the Adolescent Girls and Young Women SI contributed to promoting and protecting human rights and gender equality. The New Nets SI leverages Unitaid matching funds, thereby contributing to mobilizing increased resources.
Recommendations and Secretariat response
The evaluation recommended that:
- The SC should develop a clear, consistent, and shared definition of what ‘catalytic’ means in the Global Fund context.
The Secretariat recommends using the existing definition after incorporating the key factors and principles suggested by the evaluation.
- The SC should establish a more robust mechanism to identify a coherent set of issues for potential SI selection and prioritize them based on:
- The level of programmatic risk they pose to the achievement of the Global Fund’s SOs;
- The feasibility of addressing the identified issue and the appropriateness of the SI modality to do so; and
- The cost-effectiveness of each SI.
The Secretariat agrees with these recommendations and acknowledges the importance of SI prioritization with issues drawn from the collectively developed Global Fund Strategy. It agrees with factoring in the cost-effectiveness of each SI. Moreover, it strongly agrees considering the feasibility of addressing the identified issues and the appropriateness of the SI modality. However, the Secretariat only partially agrees with factoring in programmatic risk if the SIs are not implemented as it can be difficult to tease out the SIs’ specific incremental impacts.
- The Secretariat should continue improving the SI design process such that:
- Country-level stakeholders are actively involved in the SI design process;
- A robust TOC is included during the SI design process;
- There is an exit strategy for each SI; and
- An evaluation for each SI is incorporated in its design;
The Secretariat fully agrees with these recommendations and further indicates the need to tailor country-level consultations with the investment. It acknowledges that the inclusion of TOCs, timeframes, and milestones is already underway in the current cycle.
- The Secretariat should ensure there is a mechanism to harmonize SI activities and objectives as well as the Global Fund’s broader portfolio of country-level support.
The Secretariat agrees with this recommendation and noted there is an opportunity for greater alignment and communication with Country Teams and in-country partners.
- The Secretariat should build on its contracting, management, and oversight arrangements to ensure the appropriateness of activities being implemented and the implementer partner responsible for the SI implementation.
The Secretariat fully agrees with this recommendation and pointed that there has been remarkable effort and change in advancing the management of SIs.