Metrics from the Ground Up: Evaluating the Impact of Programs, GHIs, NGOs

What Are Data Collection Challenges at the Grassroots Level?

Creating successful health information systems in developing countries has been an extremely challenging undertaking due to many existing obstacles, including patchy funding, lack of an international infrastructure, and poor communication between independent health programs.  While global health policies endorse local supervision and integration of health information from multiple organizations, this ideal system is far from the reality.  In practice, national health systems are usually comprised of numerous independent health programs, each preserving their unique and often disorganized method of reporting.  Due to this absence of a common system, the same data is often collected and reported multiple times through separate structures, while other significant data can be lost and never reported.  In developing countries, vast inequities exist in terms of access to elements of the greater healthcare information infrastructure.  Due to these complexities and discrepancies between separately funded programs, several strategies to expand healthcare information infrastructures have been proposed.(1)

One initiative to improve global health metrics is the Health Metrics Network (HMN), a program developed in 2005 and supported by the WHO, European Union and other international agencies.  The HMN concentrates on exploring how correct and dependable information can influence informed decision-making and therefore improve health.  This initiative is founded on the idea that “information is essential for public health action: it is the foundation for policy making, planning, programming, and accountability.”(2)  Since information is sparse in developing countries, HMN is structured with the intention of increasing accessibility and application of health information.  HMN strives to construct a synchronized framework for country health information system development.(3)

Problems with Impact Evaluation Methodologies

A variety of impact evaluations exist for enterprise development projects.  In Zandniapour et. al. (2004), impact assessments for a selection of these projects are reviewed, and problems with the methodologies are assessed.  Several common flaws can be identified across these studies, including:

All of these factors serve as limitations for evaluating given program’s impact in developing countries, and stronger impact assessment methodologies are needed in order to attribute changes and draw decisive conclusions of the program’s effectiveness. (4)

Strategies for Future Impact Studies

It is crucial to improve impact evaluations for enterprise development programs.  Suggested methodologies include:

One rigorous impact assessment method is the Randomized Evaluation, which utilizes random assignment to allocate funds, track programs, or apply policies. This is a method that is strongly advocated for by the Abdul Latif Jameel Poverty Action Lab.  When conducting an impact evaluation, it is crucial to identify a comparison group who did not participate in the program but otherwise closely resembles the group who did participate.  Randomized evaluations are particularly effective in creating a valid comparison group because it does not rely on assumptions. Instead, there is an actual comparison group that is a statistical duplicate of the original.  J-PAL is comprised of a network of professors worldwide, continually conducting impact evaluations in order to investigate and expand the success of poverty reduction programs.(6)

Conclusion

Methodologically rigorous and systematic evaluations provide the opportunity to raise the impact of organizations beyond their financial contributions.  Future program planning can benefit from an understanding about past successes and failures. Therefore, reliable evaluations should be viewed as worldwide public goods.  Successful programs can be modified and used in other countries, while ineffective attempts should be discontinued.  Encouraging and financing evaluations (for example, credible randomized evaluations)(7) of effective organizations offers feedback not only for the specific organization being evaluated, but also for other contributors, governments, and NGOs.

Go To Module 4: Cost-Effective Analysis >>

Footnotes

(1) Braa et. al. “Developing Health Information Systems in Developing Countries: The Flexible Standards Strategy.”  MIS Quarterly. 31. (August 2007): 1-22. Accessed on 10 June 2010.

(2) World Health Organization, “About the Health Metrics Network.” (2010). Accessed on 10 June 2010.

(3) Ibid.

(4) Zandniapour et. al. “Review of Evaluations of Selected Enterprise Development Projects” Accelerated Microenterprise Advancement Project. (July 2004).  Accessed on 10 June 2010.

(5) Ibid.

(6) Abdul Latif Jameel Poverty Action Lab. “About J-PAL” (2010).  Accessed on 10 June 2010.

(7) Duflo, E. and Kremer, M. “Use of Randomization in the Evaluation of Development Effectiveness,” World Bank Operations Evaluation Department Conference on Evaluation and Development Effectiveness. (July, 2003). Accessed  on 10 June 2010.