Let me tell you a story

Two months ago I started to learn
more about impact measurement andat this moment,  I feel I have just scratched the surface.There
is so much more to come.

I have learnt that impact
measurement and broadly the evidence or results-oriented approach received
increasing interest in the last years in development practice. This is mainly driven
by the purpose of using the evidence gathered for learning and improvement. At
the same time, I also came across views on some challenges in doing impact
measurement. I will continue by first briefly presenting some of these challenges
and second, by introducing what I think can be an answer on how to tackle some
of them.

There is broad agreement about
the need to understand the impact of development interventions. However, doing
impact measurement also comes with it’s challenges and concerns. Parts of these
are expressed
in the views Duncan Green gathered from
a debate about the results based agenda[T1] .
For instance, some of the concerns expressed were that development practitioners face problems such as limited resources or time pressure that risks
to affect their work in promoting the change they aim for i.e. impact
the lives of people living in poverty and injustice.

image

Source: https://oxfamblogs.org/fp2p/icymi-best-of-this-summers-book-reviews-the-politics-of-evidence/#prettyPhoto

Further, the issue of “power
relations” and empowerment of beneficiaries also came up. As
Rosalind Eyben and Chris Roche argue
the risk is that a pure technical
evidence approach that treats people as subjects in need of a “treatment”
rather than citizens with a political voice might hide the structural causes
and consequences of inequity.

Some of the solutions they suggest
refer to “Engaging with a wider and more diverse group of policy actors in the
state, civil society and the private sector; whenever possible, supporting
debate, locally-driven problem solving, and independent research. It means
avoiding overly linear project-based aid modalities that demand omniscience
before they have even begun”.As it can be seen there is a recommendation to seek
ways of promoting locally-driven problem solving, more exhaustive research and engagement
with beneficiaries.

And a way of doing this, I think, is
by using a mixed methods research design for impact measurement. This is
increasingly promoted in academia in the last years and it is gradually being
adopted also by policymakers and NGOs.

There are various advantages of
using a mixed methods research design in impact measurement. A mixed methods
research gains in complementary because either quantitative or qualitative
methods may be insufficient by itself. This helps to understand and exploit
strengths and biases inherent to each method. Further, it allows the researcher
to see his work through multiple angles since quantitative and qualitative
methods provide different “pictures” and increase validity through
triangulation. Lastly but maybe most important, it includes “voice of
respondents” and has the potential to build shared values and tackle societal
issues in research community.

Here is where the Oxfam Novib’s World Citizens Panel (WCP)’s
methodology stands out, by being one of the approaches in the development
practice where mixed methods are used for impact measurement. Even more, a
particular important element of the WCP’s approach is the qualitative aspect,
dubbed “Stories of Change” and consisting in the use of the Most Significant
Change method through stories collection. This constitutes a good contribution
to using primarily or only quantitative methods, which is especially important
since quantitative methods can only measure “to what extent” change has come about but it cannot explain the
“how” and the “why”.

The main purpose of Stories of Change is to
deepen the understanding of how
change happens through Oxfam Novib’s programs. Stories of Change uncover both success stories as well as unexpected and negative
changes. This shows the impact of Oxfam Novib’s programmes but also contributes
to learning on how to improve future programmes.

One
way of looking at a story is as “recounting of events based on emotional experience
from a perspective” (see also here and in the picture
below). When involved in research,
people are used to being asked for their opinions. But they are not used to
being asked about events that happened to them. Doing the latter brings the
researcher closer to the interviewees. Also, the researcher needs to know more
than the facts. He needs to learn about how people felt during their
experiences.

image

Source: http://www.workingwithstories.org/WorkingWithStoriesThirdEdition_Web.pdf

And why collect stories, what does it bring?

Firstly, the researcher gets to
more grounded decisions about the community. Furthermore, it has a social
function because it shows that we care and want to hear people’s perspective on
the “fact”. Lastly, it allows people to voice their real views and struggles by
being able to choose what story they will tell.

“Today
is Thursday; my Friday will be a blessed one”. This is how storyteller Maryam,
whom the WCP team interviewed in Nigeria during the impact measurement of an
Oxfam Novib’s livelihoods programme, named her story. And
such stories keep reminding me why I choose to work in development. And since
“people want to tell their stories; people want to be heard” (quote from Roos
Boer, as part of the event UN at 70 referring to stories collection in Ukraine;
see more here), should
we not aim to broaden the application of this research method in impact
measurement and make tomorrow a “Friday”?

Raluca Ivanof

This post was also published on the World Citizen’s Panel website at Oxfam Novib. 

Raluca
is currently interning at the WCP team at Oxfam Novib. A student of the University
of Amsterdam, her spunky attitude and clear arguments make her a great candidate
for someone to brainstorm with. Much to our amusement, her difficult relationship
with salt stands in stark contrast with that of her recently proclaimed husband.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s