Two months ago I started to learn
more about impact measurement andat this moment, I feel I have just scratched the surface.There
is so much more to come.
I have learnt that impact
measurement and broadly the evidence or results-oriented approach received
increasing interest in the last years in development practice. This is mainly driven
by the purpose of using the evidence gathered for learning and improvement. At
the same time, I also came across views on some challenges in doing impact
measurement. I will continue by first briefly presenting some of these challenges
and second, by introducing what I think can be an answer on how to tackle some
There is broad agreement about
the need to understand the impact of development interventions. However, doing
impact measurement also comes with it’s challenges and concerns. Parts of these
in the views Duncan Green gathered from a debate about the results based agenda[T1] .
For instance, some of the concerns expressed were that development practitioners face problems such as limited resources or time pressure that risks
to affect their work in promoting the change they aim for i.e. impact
the lives of people living in poverty and injustice.
Further, the issue of “power
relations” and empowerment of beneficiaries also came up. As
Rosalind Eyben and Chris Roche argue the risk is that a pure technical
evidence approach that treats people as subjects in need of a “treatment”
rather than citizens with a political voice might hide the structural causes
and consequences of inequity.
Some of the solutions they suggest
refer to “Engaging with a wider and more diverse group of policy actors in the
state, civil society and the private sector; whenever possible, supporting
debate, locally-driven problem solving, and independent research. It means
avoiding overly linear project-based aid modalities that demand omniscience
before they have even begun”.As it can be seen there is a recommendation to seek
ways of promoting locally-driven problem solving, more exhaustive research and engagement
And a way of doing this, I think, is
by using a mixed methods research design for impact measurement. This is
increasingly promoted in academia in the last years and it is gradually being
adopted also by policymakers and NGOs.
There are various advantages of
using a mixed methods research design in impact measurement. A mixed methods
research gains in complementary because either quantitative or qualitative
methods may be insufficient by itself. This helps to understand and exploit
strengths and biases inherent to each method. Further, it allows the researcher
to see his work through multiple angles since quantitative and qualitative
methods provide different “pictures” and increase validity through
triangulation. Lastly but maybe most important, it includes “voice of
respondents” and has the potential to build shared values and tackle societal
issues in research community.
Here is where the Oxfam Novib’s World Citizens Panel (WCP)’s
methodology stands out, by being one of the approaches in the development
practice where mixed methods are used for impact measurement. Even more, a
particular important element of the WCP’s approach is the qualitative aspect,
dubbed “Stories of Change” and consisting in the use of the Most Significant
Change method through stories collection. This constitutes a good contribution
to using primarily or only quantitative methods, which is especially important
since quantitative methods can only measure “to what extent” change has come about but it cannot explain the
“how” and the “why”.
The main purpose of Stories of Change is to
deepen the understanding of how
change happens through Oxfam Novib’s programs. Stories of Change uncover both success stories as well as unexpected and negative
changes. This shows the impact of Oxfam Novib’s programmes but also contributes
to learning on how to improve future programmes.
way of looking at a story is as “recounting of events based on emotional experience
from a perspective” (see also here and in the picture
below). When involved in research,
people are used to being asked for their opinions. But they are not used to
being asked about events that happened to them. Doing the latter brings the
researcher closer to the interviewees. Also, the researcher needs to know more
than the facts. He needs to learn about how people felt during their
And why collect stories, what does it bring?
Firstly, the researcher gets to
more grounded decisions about the community. Furthermore, it has a social
function because it shows that we care and want to hear people’s perspective on
the “fact”. Lastly, it allows people to voice their real views and struggles by
being able to choose what story they will tell.
is Thursday; my Friday will be a blessed one”. This is how storyteller Maryam,
whom the WCP team interviewed in Nigeria during the impact measurement of an
Oxfam Novib’s livelihoods programme, named her story. And
such stories keep reminding me why I choose to work in development. And since
“people want to tell their stories; people want to be heard” (quote from Roos
Boer, as part of the event UN at 70 referring to stories collection in Ukraine;
see more here), should
we not aim to broaden the application of this research method in impact
measurement and make tomorrow a “Friday”?
This post was also published on the World Citizen’s Panel website at Oxfam Novib.
is currently interning at the WCP team at Oxfam Novib. A student of the University
of Amsterdam, her spunky attitude and clear arguments make her a great candidate
for someone to brainstorm with. Much to our amusement, her difficult relationship
with salt stands in stark contrast with that of her recently proclaimed husband.