The presentation titled ‘Observing from space when evaluators cannot observe in the field’ by Joachim Vandercasteelen discussed how georeferenced data can be used for project evaluations when evaluators cannot travel to the field. The presentation provided insights from a project evaluation that the IEG is currently conducting on an environmental project in Madagascar.
Session Summary:
Firstly, geospatial data allows us to accurately measure an indicator of project effectiveness, which in the case of this project is the reduction in the change in deforestation rates. Secondly, by collecting information before and after the project, and for locations supported and not supported by the project, the assessment of project effectiveness becomes very robust. Thirdly, exploring several geospatial dimensions of effectiveness heterogeneity helps to better understand the role of behavioral factors in reducing deforestation. A multidisciplinary approach that combines geospatial data with qualitative information provides a solid methodology to robustly assess project effectiveness, as well as to conjecture the “why” of project effectiveness.
The Session’s Key Messages:
- Independent project evaluation is at the core of IEG’s mandate, and IEG is exploring innovative approaches to the methods of project evaluation.
- When travel restrictions do not allow business-as-usual field visits for project evaluation, georeferenced data offer a unique opportunity.
- Evaluators can collect and analyze geospatial data available at no cost and over a series of time frames that help in identifying research questions and appropriate methodology.
- IEG has been using geospatial data for the evaluation of an environmental project in Madagascar in multiple ways:
- A geospatial indicator of project effectiveness has been constructed
- Geospatial data allows us to explore and visualize spatial and temporal variation in indicators of project effectiveness
- This variation in geospatial data allows for the robust assessment of project effectiveness using a difference-in-difference approach
- It allows us to explore heterogeneity in project effectiveness, to better understand the “why” of project effectiveness
- Thus, georeferenced data can be used in a multidisciplinary method that allows for a robust quantitative assessment of project effectiveness and exploration of the qualitative assessment of the “why” of project effectiveness.
Joachim’s Answers to Participants’ Questions:
- It would be interesting to hear what approaches and geo-data could be used to measure the potential spatial access to urban facilities (health services for instance):
- Geo-data: Open Street Maps offers very good (partly citizen science-based) information on the location and type of urban facilities around the world.
- Do you have a recommendation on tools for data analytics?
- This is a very general question, but a lot of freeware is available: Qgis for geospatial data visualization and analysis, or Python for big data management and simplistic geospatial operations.
- Geospatial data is indeed interesting – but oftentimes the data is hard to find or is private. Are there open data sources that you can recommend?
-
- There are many sources of public georeferenced data: NASA satellite data on land cover and derived products (https://search.earthdata.nasa.gov/search), FAO’s Hand-in-hand platform for agroecological data (https://data.apps.fao.org/), UN datasets on biodiversity indicators (https://www.unbiodiversitylab.org/), etc.
- Do big data and geospatial methods lend themselves well to the work of small NGOs who work on behavior change at a much smaller scale? For example, personal hygiene, sexual and reproductive health, civic participation?
- Obtaining qualitative information on human behavior is more difficult, especially on health or personal attitudes. However, one can still try to come up with a proxy for human behavior after reviewing the academic and policy literature on the topic. For example, in the evaluation of the environmental project in Madagascar, we proxy households change in the use of shifting cultivation (that requires cutting down forests) by the change in forest fires over time. But obviously, geospatial data is not a silver bullet and will not allow us to proxy all human behaviors.
Further resources:
Check out Joachim’s blog for the World Bank for more information here.
Disclaimer: The opinions expressed in this article are the author’s own and do not necessarily reflect those of the European Evaluation Society.
The geospatial its very necessary in ways of saving our environment with aims of making green world and better living condition of world’s population so, environment protection is desirable to day from deforestation.
I want to know about dependency on secondary data and pixels during emergency period.
Dear Joachim,
When I blue-water sailed (professionally) fifty years ago I collected geospatial data (although the term didn’t yet exist) by observing the sun, moon, other planets and stars, landmarks, clock set to GMT, through a geospatial data collection instrument called a sextant. The data were quantitative, i.e. angles & time. These data allowed me to plot my position on a chart, i.e. to transfer the data to a physical model of reality. I would repeat the exercise a few times a day and at least everyday at noon local. Spatial and temporal variations between quantitative data were made meaningful by translating them into qualitative data on the chart, i.e. you could see on the model where you were, where you had come from and the distance covered.
Other quantitative data such as elapsed time of variation would give an estimate of average speed. These data would allow for an estimation of ETA at waypoints, taking into account personal intuitive estimates of current, wind direction & velocity, and possible use of engine. On a few occasions I came across local fishing folk (some in dugouts 2-3 days from shore) who would give me verbal advice (qualitative data) about how best to navigate to where I wanted to go. Those qualitative data trumped my calculations, plots, etc. each and every time. In other words, context appropriate knowledge is most valuable.
The only difference today, as is the case with your Madagascar example, is that, instead of a model (and all models are wrong) we have photos of the real thing (I have hit reefs that were not on the chart ). And this is of course a key improvement.
Having said this, the rest remains pretty much the same. Interpretation, estimation and judgment, all of which incorporate and express a set of values, or if you wish, a world view, determine at the end of the day the value of the process.
Distinguishing between “a robust quantitative assessment of project effectiveness” and “exploration of the qualitative assessment of the “why” of project effectiveness” appears to me, quite frankly, somewhat obtuse and biased towards the use of numbers and their manipulation. These kinds of statements, that I come across regularly, tend to be ideological and/or market driven, and, unfortunately, they do the exact opposite of what their proponents think, which is to reduce the credibility of the overall message.
I also took a look at your WB blog. There’s a lot there that I would challenge in terms of methodology, however, to be fair, I realise that it’s a high-level summary and that there’s undoubtedly a lot more serious work and thought behind it. Having said this, the kernel of insight resides in your statement: “These factors are often highly contextual and linked with human behavior, which is much more difficult to measure from space.” It is indeed all about messy human beings, who in different and varying collectivities and contexts, incorporate the attributes of complexity.
Finally, as I state repeatedly in various fora, if it doesn’t value it is not an evaluation: The extent to which something has been achieved is a measurement question, not a valuing question. Checking to see whether intended results have been achieved, or not, unintended positive, negative, etc. does not make an evaluation. Evaluative reasoning is required to value.
Young (er) evaluation professionals such as you are key to changing the trajectory of mainstream evaluation practice so that it effects positive change for living systems on this planet. I salute your engagement and willingness to share your perspectives, and I encourage you to keep it up. Discussion, debate, self-reflection and deep thinking are also key to keeping us alive and well.
Thank you for engagement and please don’t hesitate to get in touch with me directly should you so wish.
Best regards,
Ian C. Davies
Credentialed Evaluator