Designing and measuring the progress and impact of health research capacity strengthening initiatives

Strengthening health research capacity in low and middle income countries (LMICs) is one of the most effective ways of advancing their health and development. Over the next 10-20 years the current trend towards more emphasis on capacity strengthening within health research projects is likely to accelerate. This is because lack of capacity to utilise research findings, to integrate them into health systems and embed them in institutions, is a major bottleneck in our ability to maximize the benefits of research for development.

Definition of research capacity strengthening - By health RCS we mean a ‘process of individual and institutional development which leads to higher levels of skills and greater ability to perform useful research.’

Health RCS takes many forms and operates at three different levels, often simultaneously – individual (e.g. PhD training), institutional (e.g. establishing research governance systems) and national/international (e.g. international collaborations). The complexity and heterogeneity of RCS initiatives[2] makes it difficult to evaluate their effectiveness.

Our Capacity Research Unit at LSTM does research to inform better design and measurement of RCS programmes. From our work with health research funders, research institutions and national research systems, and through our own experience of managing capacity strengthening programs in LMICs we have developed an evidence-based 5-step approach to designing RCS programmes which is applicable to diverse contexts. We have collated information about different approaches to RCS and summarized what works best in which type of context and why. We have analysed evaluators' reports to bring together a comprehensive list of indicators that have been used to measure RCS processes and impact.

Our research has inevitably unearthed some critical priorities, challenges and tensions that need to be addressed over the next 10-20 years to ensure we make significant progress in bridging the gap between research generation and population health and development in LMICs. Integration of different research disciplines, as well as enhanced research capacity, is essential to make a significant impact on strengthening health systems and to support effective policies and decision-making. In this paper we outline some of the key findings of our research and highlight what has emerged from our research as priorities for the next 10 years.

Evidence to guide the design and monitoring of RCS efforts in LMICs is limited and formulating a general approach is complex because each programme is unique. Capacity strengthening programmes usually start off with a ‘needs assessment’ but by doing this, two critical steps are missed out – defining the expected outcomes and identifying the capacity needed to achieve the outcomes. Both these steps are important in developing a robust and comprehensive action plan and in promoting ownership and sustaining change. Using published evidence, including literature on organisational management, we have developed a five step pathway for designing and evaluating RCS programmes and tested it in a variety of contexts in Africa[3–5].

The pathway starts with a clear goal and objectives, and it make the capacity required to achieve the goal explicit. We derive a description of the optimal capacity needed to achieve the goal from a comprehensive search of the literature and include inputs by international experts relevant to the goal of the RCS, and a ‘reality check’ by LMIC partners. The unique aspect of this 5-step approach is the production of this evidence-based ‘benchmark’ (i.e. definition of optimal capacity needed to achieve the goal) which is designed specifically for each project and against which existing capacity is compared. This approach differs from most needs assessments of research capacity which do not use a rigorous benchmark and therefore are not able to provide reassurance that significant areas have not been overlooked.

Using the optimal capacity ‘benchmark’ we identify capacity gaps and develop the action plan to fill the gaps in collaboration with LMIC partners. The plan is designed to be flexible enough to generate and utilise ongoing learning; strategies for promoting sustainability are incorporated from the outset. The plan takes into account not only technical, managerial and financial processes within an organisation, but also the individuals who work in the organisation and the wider system within which the organisation operates and is financed and managed. Interestingly we have consistently found when using this five-step approach in practice, that 60-70% of activities in the plan are ‘no cost’. For example, establishing a new committee, drafting protocols and strategies, or providing in-house training.

Understanding what indicators or metrics can be used for health RCS and their appropriateness, or not, in different contexts, is critical to monitor the performance of RCS programmes, to measure achievement and to demonstrate accountability. Ideally RCS funders would like a few common measurable, reliable indicators for measuring RCS but they also need to demonstrate project impact on, for example, policies and poverty, for which direct attribution is difficult to prove. Our research has shown that the majority of evaluations of health RCS meet very few of the generally accepted quality standards. Most of the evaluations we have reviewed were retrospective so they lack comparative baseline data, and the purpose and indicators for outcomes and impact were often not stipulated.

We were unable to find information in the published literature about indicators for RCS that were SMART (Specific, Measurable, Attainable, Realistic and Time Bound). We therefore collaborated with a group of global health research funders and collated all the indicators we could find in their evaluation reports of RCS evaluations[6]. We extracted the indicators into a matrix across individual, institutional and national/international levels. Critical to making sense of RCS outcomes is the need to be explicit about the pathway by which change is to be brought about (i.e. the theory of change) and to use indicators that reflect different stages on the pathway. We therefore synthesized indicators across potential impact pathways (activities to outputs to outcomes) and iteratively verified our findings with key health RCS evaluation stakeholders. The indicators we identified were primarily of activities, outputs or outcomes with a strong bias towards the more easy-to-measure individual level indicators. There was very little information on the inter-relationships along a ‘change pathway’ so measurements of progress in developing research capacity over time were scarce.

RCS is a complex process with a long intervention pathway and examples of good quality RCS evaluations are hard to find. Such evaluations involve theory-driven evaluative thinking, identification of underlying assumptions and an explicit rationale for any actions[7]. It is important to consider long-term impact across different levels throughout the whole project cycle using clear conceptual frameworks, multiple data sources and valid standards to enhance quality.

RCS evaluations should incorporate a theory of change as this helps to identify impact trajectories, strengthen evaluation rigour, foster assessment of generalisability to other contexts, and guide influence on policy and practice. However, most evaluations currently do not have time and resources to incorporate theory-informed indicators of impact and sustainability, or to collect data against these indicators. This means opportunities to enhance knowledge and learning among funders and funding recipients about how to improve planning, monitoring and evaluation of health RCS initiatives are being missed[8].

Some funder groups (e.g. UK Collaborative on Development Sciences, the ESSENCE on Health Research Initiative) are now coming together to share experiences and learn from health RCS initiatives but they are hampered by a lack of common approaches and of systematic methods for documenting RCS. Ultimately, timely sharing information about RCS will help to avoid repeating mistakes, and enhances knowledge about how to do better health RCS. Our research highlighted some key challenges concerning learning from RCS efforts that will need to be addressed in future. Health RCS activities are often ‘bolted on’ to research projects but expectations that researchers themselves will spend a significant amount of time documenting and using learning to improve the RCS activities at the same time as delivering their primary research, may be unrealistic. Research funders face a trade-off between their need to show accountability and value for money, and the extent to which they should invest in facilitating and sharing learning.

Our research suggests that funding recipients and other stakeholders should be actively involved in all stages of the RCS evaluation process as this ‘empowerment’ makes RCS initiatives more likely to become self-sustaining[9]. It also helps to detect and correct problems early, makes the decisions underlying the evaluations more visible, and encourages sharing and use of the evaluation results. Stakeholders such as funding recipients have greater in-depth knowledge about the project and its context compared to external evaluators, which is vital for solving problems and sustainability. However there is a tension between this approach, which takes time and needs an additional, ear-marked budget, and the traditional summative model of evaluation carried out by an independent external team, which many funders' perceive as better value for money, for demonstrating accountability and for producing rapid results[10].

10 year perspective on using learning from research to design more effective RCS programmes