![[Feedback]](/icons/banner/feedbackACT.gif){kind=icon}
![[For Subscribers]](/icons/banner/subscriberACT.gif){kind=icon}
![[Archive]](/icons/banner/archiveACT.gif){kind=icon}
![[Search]](/icons/banner/searchACT.gif){kind=icon}
![[Contents]](/icons/banner/tocACT.gif){kind=icon}
|
|
|
|
|
||
3 Senior Child Survival Advisor, Save the Children, Westport, CT 06880; 4 Monitoring and Evaluation, Save the Children, Washington, DC 20036; and 5 Africa Regional Health Advisor, Save the Children, Pretoria 0127, South Africa
* To whom correspondence should be addressed. E-mail: dmarsh{at}savechildren.org.
 |
ABSTRACT |
|---|
 TOP ABSTRACT LITERATURE CITED |
|---|
Programmers and researchers have different agendas, serve different stakeholders, and use different terminologies. Yet, to answer international health's most vexing challenges, they must and do collaborate. Results frameworks are simple models that both disciplines can understand and use. The United States Agency for International Development adopted the results framework to strengthen the planning and evaluation of its projects (1,2). Despite a decade of growing use among the Agency and its partners, relevant literature is scant. Toffolon-Weiss, Bertrand, and Terrell (3) explain it and provide a case study of its use in an HIV/AIDS initiative in Central America. Others cite it as an example of a program pathway (4) or a model for developing large-scale intervention and assessment programs (5).
Results frameworks can serve a range of social programs, agencies, and levels. Here we restrict our attention to a "6-box" version commonly used by health programmers (Fig. 1) (6). Each box reflects a specific category of programmatic result, all of which contribute to the overall goal of improved health status, typically reduced mortality, morbidity, or fertility. The framework serves 2 critical programmatic functions. By describing the steps toward a program's ultimate goal, it indicates the general programmatic direction to achieve that goal through stepwise results. Second, by outlining a hierarchy of results, it prompts managers to specify indicators to track progress toward achieving them.
|
The bottom tier of the framework has 4 intermediate results (IR), which are essential steps toward achieving the SO. They are: IR-1, access to and availability of health services increased; IR-2, quality of health services improved; IR-3, demand for health services or behaviors increased; and IR-4, social and policy environment enabled. Access and availability refer to the likelihood that a person and a service will actually meet (through the person going to the service, the service going to the person, or meeting midway) and that the necessary supplies and services will be available. Access refers not only to geography but also to socioeconomic factors that can serve as barriers to care. The concept of quality is complex and refers not only to the technical quality of the services but also to clients' perceptions of quality and acceptability—not always the same as technical quality. Nonetheless, good perceived quality facilitates use. The first 2 IR comprise "supply." IR-3, demand, includes the behavioral determinants of using the service or practice, i.e., knowledge, skills, norms, attitudes, beliefs, and the like. IR-4, an enabled environment, facilitates high demand and high supply. Environmental factors are important at all levels, from the community (i.e., its mobilization or capacity or social norms) to the capital city (i.e., favorable policies).
Within the results framework, the SO is the level that relates to the use of interventions (the "what": what medicine, vaccination, commodity, or practice we want beneficiaries to use). The IR are important steps to achieve through the chosen delivery strategies (the "how": how we propose to increase use). Finally, the goal is the "why": why we design the program.
The results framework has limitations, mainly because of its apparent simplicity. As mentioned, quality has many dimensions. Actual, as opposed to perceived, technical quality would be an effect modifier between use and goal, necessitating an additional box and a divided concept. An enabled environment is usually an effect modifier of program activities (not shown in the framework) to achieve the other 3 IR. The results framework omits external environmental factors (apart from those in IR4) that can ease or constrain achieving the results and that are beyond the reach of programmers, factors as diverse as a new road or a tsunami. Finally, the framework lacks the operational details (such as those found in logical frameworks) that managers and some donors need; however, standard detailed implementation and monitoring plans that are based on the framework provide these. In summary, the results framework is not perfect, but it gets enough "big ideas" right to be useful.
Indeed, the advantages are many. 1) This common framework is simple enough to be easily understood by programmers and researchers, who can, in turn, use it to communicate with their constituents. 2) Boxes connected by directional arrows indicate hypotheses, either the programmer's developmental assumptions or the researcher's hypothesis. 3) Programmers benefit from a model that applies throughout the program cycle, from situation analysis to design to evaluation. 4) Programmers and scientists benefit from a model that focuses on life-benefiting results for people. 5) Finally, the model, an approximation and oversimplification of reality, stimulates questions that are relevant to programs and ultimately to human health.
Applications and research questions
The results framework melds 2 disciplines: discovery science and delivery science. That is, the association between the SO and the goal rests on the efficacy studies of "discovery scientists" to assess the impact of an intervention on a health outcome. On the other hand, the association between the IR and the SO rests on common-sense assumptions that "delivery scientists" need to test and quantify. Below we propose 3 sets of questions arising from the framework: 1) from the framework itself, 2) from the framework applied to a specific program, and 3) from the framework applied to a research initiative.
The framework itself. Two general questions arise directly from the generic framework, regardless of the specific program setting. First, what factors might modify the expected association between supply and demand versus use or between use versus health? Victora and coworkers have proposed groups of such effect modifiers: 1) behavioral (provider, beneficiary, others); 2) biological (antagonism, synergy, curvilinear dose-response, limited scope for improvement, missing cofactor); and 3) contextual (socioeconomic, demographic, environmental, baseline health, health system, other programs) (9,10).
The second general question relates to optimizing the mix of strategic investments to strengthen supply, demand, and enabling environment. An exclusive investment in only 1 strategy may not increase use. For example, investing only in IR-1 might yield highly available interventions, poorly delivered, with incomplete demand, whereas investments limited to IR-2 could yield incompletely available interventions, well delivered (where available), with incomplete demand. Programmers must address gaps in IR, although all may not need attention. So how should effort be apportioned, and what is the evidence?
Program application. An example of a health program designed for an extremely impoverished setting, Southern Sudan, highlights other questions. A situation analysis (qualitative and quantitative) characterized the environment, including the health system; the causes of illness and death; and the levels of indicators of use, availability, demand, and so on—and the gaps. This analysis, in turn, directly informed the selection of life-saving interventions (7,8). More difficult was the question of how to deliver these interventions, especially because the health system was almost nonexistent.
We drafted menus of strategies to achieve each IR (Table 1), with substrategies for each (not shown; available on request). For example, improving access through facility-based services includes renovating existing facilities; constructing new facilities; expanding service packages; integrating new interventions into existing packages; staffing up to or beyond plan; increasing service hours per day or days per week; implementing insurance, loan, or waiver schemes; using local language; respecting cultural norms; installing radio or telephone; using, controlling, and maintaining an ambulance; engaging community-based providers; and eliminating other barriers, such as removing mines or unexploded ordinance and repairing roads or bridges.
|
In most settings, working through existing systems is a sensible starting point, at least to reach some of the population. But most countries, even those with adequate systems, have geographic areas or populations beyond their system's reach. What exactly are the options for increasing access, availability, quality, etc. to deliver interventions to those outside the existing system? Examining the status of the "fifth quintile" (11) is an increasingly common way to quantify this inequity. Reaching the bottom quintile (or tertile or decile) requires fresh thinking and local adaptation.
A catalogue of major and minor delivery strategy options is helpful, but it forces additional questions: How complete is the list? Which strategies are best for a given epidemiological setting (12)? Ecologic setting? Health systems setting? Cultural setting? Socioeconomic setting? Which strategies are best for given interventions or packages of interventions? Which strategies best serve equity? At what cost? At what cost/benefit?
Refining research. The generic results framework is a useful point of departure for both programmers and researchers. We recently began examining IR-4, ultimately to characterize the interaction between increasing the use of health interventions and increasing community capacity. To stimulate thinking, we placed 2 very simplified frameworks side by side: a results framework for health outcomes and a social change model (13) for community capacity outcomes (Fig. 2). The following questions arose relating to both processes and the outcomes of community capacity. What activities result in the most community capacity? How much do health program activities influence community capacity? How much do community capacity activities influence individual health outcomes? How do community capacity activities modify the effect of health activities on achieving individual outcomes? How does community capacity itself modify the effect of health program activities on achieving individual health outcomes? Additional questions, although not informed by Figure 2, derived from the foregoing questions. What are the most important external factors that help or hinder these associations? How much does community capacity help sustain individual outcomes? Which activities are associated with which domains of community capacity? Which strengthened domains of community capacity are associated with which specific or type of individual health outcome?
|
Development is complex, even when the interventions are "silver bullets" such as vitamin A supplementation or measles vaccination. A results framework, admittedly (and fortunately) a gross oversimplification of reality, brings advantages. It poses key questions for programmers and researchers by an orderly arrangement of critical health system issues and even by what it omits (Fig. 3)! Thus, a thoughtful consideration of the framework can inform a response to Scheirer, who argued that the performance-monitoring literature, including that related to results frameworks, failed to assess relations among performance measures, i.e., among IR or between inputs and IR (14). Similarly it can inform a response to Bollen, Paxton, and Morishima, who urged USAID and others to carefully characterize confounders before drawing conclusions from the more commonly measured inputs and results (15).
|
The global health community has mature, productive, respected research institutions and teams to develop new interventions, sometimes called "discovery science." On the other hand, the discipline to inform how to bring these discoveries to families who need them, sometimes called "delivery science," is far less advanced. Too few people in developing countries currently receive the evidence-based health interventions that could save their lives (16). Indeed, at the 2005 World Health Assembly, Bill Gates prioritized the need for "more thinking and funding to delivering interventions—not just discovering them" (17). A results framework can convene delivery science partners (programmers and researchers) as well as advocates, policy makers, and donors around a simple model that asks tough questions. The answers to those questions may bring the Millennium Development Goals within reach.
|
ACKNOWLEDGMENTS |
|---|
|
FOOTNOTES |
|---|
2 Author disclosures: D. R. Marsh, J. C. Alegre, and K. Z. Waltensperger, no conflicts of interest.
|
LITERATURE CITED |
|---|
|
TOPABSTRACTLITERATURE CITED |
|---|
1. United States Agency for International Development, Center for Development Information and Evaluation, Bureau for Policy and Program Planning. Resource book on strategic planning and performance monitoring under reengineering. Washington, DC: U.S. Agency for International Development; 1996.
2. United States Agency for International Development, Center for Development Information and Evaluation, Bureau for Policy and Program Planning. Performance monitoring and evaluation tips. Washington, DC: U.S. Agency for International Development; 1997.
3. Toffolon-Weiss MM, Bertrand JT, Terrell SS. The results framework—an innovative tool for program planning and evaluation. Eval Rev. 1999;23:336–58.
4. Wind L, Brooks D. Child welfare demonstration projects: a model for implementation and evaluation. Child Youth Serv Rev. 2002;24:543–58.
5. Epstein J, Collins KK, Pancella T. The Doubles—a case study on developing a technology-based substance abuse education curriculum. Eval Rev. 2004;28:539–63.
6. United States Agency for International Development, Office of Sustainable Development, Bureau for Africa. Health and family planning indicators: a tool for results frameworks, Volume 1. Washington, DC: U.S. Agency for International Development; 1999 (
http://sara.aed.org/publications/cross_cutting/indicators/html/indicators1.htm
).
7. Jones G, Steketee RW, Black RE, Bhutta ZA, Morris SS, and the Bellagio Child Survival Study Group. How many child deaths can we prevent this year? Lancet. 2003;362:65–71.[CrossRef][Medline]
8. Darmstadt GL, Bhutta ZA, Cousens S, Adam T, Walker N, de Bernis L, Lancet Neonatal Survival Steering Team. Evidence-based, cost-effective interventions: How many newborn babies can we save? Published online March 3, 2005; http://image.thelancet.com/extras/05art1217web.pdf.
9. Victora CG, Habicht JP, Bryce J. Evidence-based public health: moving beyond randomized trials. Am J Public Health. 2004;94:400–5.
10. Victora CG, Schellenberg JA, Huicho L, Amaral J, El Arifeen S, Pariyo G, Manzi F, Scherpbier RW, Bryce J, Habicht JP. Context matters: interpreting impact findings in child survival evaluations. Health Policy Plan. 2005;20: Suppl 1:i18–31.[Abstract]
11. Victora CG, Fenn B, Bryce J, Kirkwood BR. Co-coverage of preventive interventions and implications for child-survival strategies: evidence from national surveys. Lancet. 2005;366:1460–6.[CrossRef][Medline]
12. Black RE, Morris SS, Bryce J. Where and why are 10 million children dying every year? Lancet. 2003;361:2226–34.[CrossRef][Medline]
13. Figueroa ME, Kincaid DL, Rani M, Lewis G. Communication for social change: an integrated model for measuring the process and its outcomes, The Communication for Social Change Working Paper Series: No. 1., New York: The Rockefeller Foundation; 2002.
14. Scheirer MA. Getting more "bang" for your performance measures "buck.". Am J Evaluation. 2000;21:139–49.
15. Bollen K, Paxton P, Morishima R. Assessing international evaluations: an example from USAID's democracy and governance program. Am J Evaluation. 2005;26:189–203.
16. Bryce J, El Arifeen S, Pariyo G, Lanata CF, Gwatkin D, Habicht JP, Multi-Country Evaluation of IMCI Study Group. Reducing child mortality: can public health deliver? Lancet. 2003;362:159–64.[CrossRef][Medline]
17. World Health Organization. Remarks of Mr. Bill Gates, cofounder of the Bill and Melinda Gates Foundation, at the World Health Assembly, www.who.int/mediacentre/events/2005/wha58/gates/en/print.html, 5/16/2005.
This article has been cited by other articles:
|
|
 |
|
  J. L. Leroy and P. Menon From Efficacy to Public Health Impact: A Call for Research on Program Delivery and Utilization in Nutrition J. Nutr., March 1, 2008; 138(3): 628 - 629. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. L. Garrett Improving Results for Nutrition: A Commentary on an Agenda and the Need for Implementation Research J. Nutr., March 1, 2008; 138(3): 646 - 650. [Abstract] [Full Text] [PDF]
|
|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
