Case Study: Observing from space when you cannot observe from the field

Search on EES

 

The presentation titled ‘Observing from space when evaluators cannot observe in the field’ by Joachim Vandercasteelen discussed how georeferenced data can be used for project evaluations when evaluators cannot travel to the field. The presentation provided insights from a project evaluation that the IEG is currently conducting on an environmental project in Madagascar. 

Session Summary:

Firstly, geospatial data allows us to accurately measure an indicator of project effectiveness, which in the case of this project is the reduction in the change in deforestation rates. Secondly, by collecting information before and after the project, and for locations supported and not supported by the project, the assessment of project effectiveness becomes very robust. Thirdly, exploring several geospatial dimensions of effectiveness heterogeneity helps to better understand the role of behavioral factors in reducing deforestation. A multidisciplinary approach that combines geospatial data with qualitative information provides a solid methodology to robustly assess project effectiveness, as well as to conjecture the “why” of project effectiveness.

The Session’s Key Messages:

  • Independent project evaluation is at the core of IEG’s mandate, and IEG is exploring innovative approaches to the methods of project evaluation.
  • When travel restrictions do not allow business-as-usual field visits for project evaluation, georeferenced data offer a unique opportunity.
  • Evaluators can collect and analyze geospatial data available at no cost and over a series of time frames that help in identifying research questions and appropriate methodology.
  • IEG has been using geospatial data for the evaluation of an environmental project in Madagascar in multiple ways:
    • A geospatial indicator of project effectiveness has been constructed
    • Geospatial data allows us to explore and visualize spatial and temporal variation in indicators of project effectiveness
    • This variation in geospatial data allows for the robust assessment of project effectiveness using a difference-in-difference approach
    • It allows us to explore heterogeneity in project effectiveness, to better understand the “why” of project effectiveness
  • Thus, georeferenced data can be used in a multidisciplinary method that allows for a robust quantitative assessment of project effectiveness and exploration of the qualitative assessment of the “why” of project effectiveness. 

Joachim’s Answers to Participants’ Questions: 

  • It would be interesting to hear what approaches and geo-data could be used  to measure the potential spatial access to urban facilities (health services for instance):
    • Geo-data: Open Street Maps offers very good (partly citizen science-based) information on the location and type of urban facilities around the world.
  • Do you have a recommendation on tools for data analytics? 
    • This is a very general question, but a lot of freeware is available: Qgis for geospatial data visualization and analysis, or Python for big data management and simplistic geospatial operations.
  • Geospatial data is indeed interesting – but oftentimes the data is hard to find or is private. Are there open data sources that you can recommend?
  • Do big data and geospatial methods lend themselves well to the work of small NGOs who work on behavior change at a much smaller scale? For example, personal hygiene, sexual and reproductive health, civic participation?
    • Obtaining qualitative information on human behavior is more difficult, especially on health or personal attitudes. However, one can still try to come up with a proxy for human behavior after reviewing the academic and policy literature on the topic. For example, in the evaluation of the environmental project in Madagascar, we proxy households change in the use of shifting cultivation (that requires cutting down forests) by the change in forest fires over time. But obviously, geospatial data is not a silver bullet and will not allow us to proxy all human behaviors.  

Further resources:

Check out Joachim’s blog for the World Bank for more information here.

 

Disclaimer: The opinions expressed in this article are the author’s own and do not necessarily reflect those of the European Evaluation Society.