A Non-Expert at an Expert Meeting : Methodology, Measurement… Merlot?
* All starred words are
explained in the ‘Glossary of Expertise’ at the end
International Women’s Day is
celebrated around the world with dancing; singing; eating and celebrating gender
equality achievements globally and making steps towards future parity. However,
in the Oxfam Novib Impact Measurement* Team, the 8th March was
instead a day for celebrating methodology*; measurement; evaluation and impact at
Humanity House in the centre of Den Haag. As the new intern on the
quantitative* side of the team I was invited to attend the day.
It began at 8.30 as the team
gathered in the building, finding projectors and flipcharts and welcoming
people as they ventured into the basement where the plenary sessions would take
place. My Dutch vocabulary expanded as I directed people to the cloakroom:
“garderobe” and for tea and coffee “thee of koffie?”, although my incapacity to
respond to any questions more complicated than “where are the toilets?” quickly
gave me away as a non-Dutch imposter. I snuck into the back of the opening
session and learnt about the necessity of failure for learning – should we have
a mandatory level of failure*? Other
themes that reoccurred throughout the day were questions such as: how do theory
and practice relate? How to involve key stakeholders*? How to create ownership*
of evaluations? How to learn from results?
Photo credit: Tessa Moolenaar
At the back of the plenary
session I was able to nod along and take notes as if I too was an expert but in
the breakout sessions this changed. After hiding on the last row during the
presentation on an initiative called Balika trying to reduce child marriage in
Bangladesh (which was a mistake as I am only 158cm tall…) we split up into
smaller groups where the real experts were able to critically reflect on the
presentation and results. I realized very quickly how little I knew about
development projects – I had no idea about the issue of child marriage (this
project aimed at improving the skill level of girls so they were perceived as
an asset and not a liability); I had no idea about what are considered “good”
results (apparently a change of 20% in 18 months is a surprisingly high result)
and I had no idea about the differences between, outputs; outcomes and impact
(please see glossary). The learning curve was steep but the climb was aided by
the supply of biscuits, tea, coffee, and, of course, lunch.
Photo credit: Tessa Moolenaar
Lunch was one of the highlights
of the day: it was an incredible feast of salads and sandwiches and little
sweet treats. It was also an opportunity to meet with some of the ‘experts’,
who were very willing to share their work and experiences. I met one woman who
had spent a year in a Nigerian village growing her own food to exchange with
her neighbours (there was a roadblock by armed rebels, preventing any food from
reaching the village) during her time working on a local finance
initiative. Everyone could also “run like a refugee” in
Humanity House’s immersive experience which I did, ending the experience by
pulling a trigger on some of the experts who had “run” earlier – something I
would not like to repeat in reality!
Photo credit: Tessa Moolenaar
The second breakout session
simply confirmed what the morning had already taught me: that my expertise lies
more in drinking tea than in impact measurement. However, the presentation from
OxfamGB* really interested me. Their emphasis on investing in skills of the
people in partner organisations and giving back to the local community caught
my attention. For example, the data is immediately made available to the
community once everyone has been interviewed – allowing them to see, for
example, how many goats the community has, or what is the average perception of
the occurrence of GBV*? I think that skills investment is key to creating
sustainable change and giving back means the interview relationship is not
abused by taking too much time and knowledge without returning anything.
The day ended with one final get-together,
exchanging the last pieces of advice and conducting an online survey (it was an
impact measurement event after all…) to see which pieces of advice we
collectively liked the most. The final
part of the day is where my expertise came in useful – drinking red wine, eating
snacks and chatting to new people at a borrel*.
Glossary of Expertise
Borrel – Dutch word to
describe drinks and chatting, normally accompanied by snacks such as meatballs
and bitterballen. It is gezellig!
GBV – an acronym of Gender
Based Violence; “Gender-based violence
involves men and women, in which the female is usually the target, and is
derived from unequal power relationships between men and women.” It is often
used interchangeably with the term “Violence against women”.
Impact – contested
definition but most people consider it a long-term result of a project (both
positive and negative) such as higher income or greater employment opportunities.
It is often derived using outputs and outcomes but is much harder to
Impact measurement – the
measurement of the impact of various projects. Not to be confused with MEAL (or
MEL) – monitoring, evaluating and learning which is similar, but often more
Mandatory level of failure
– for example this might be a rule that states that 20% of all projects have to
be failures; the motivation behind this is to allow honesty and greater
Methodology – the process
of how the research is carried out, so for instance it could be surveys and
then followed up with interviews, or it could be participatory action research.
Outcome – the result of
an output, so for instance an increase in the number of children attending
Output– a countable
change eg the number of schools built.
Ownership – the idea that
project leaders/beneficiaries/stakeholders need to be involved in all aspects
of the project so they feel the results are applicable to them and they can use
them, so it is not just an external requirement.
OxfamGB – the biggest
Oxfam affiliate with a different method of carrying out impact measurement to
Oxfam Novib, these are known as Effectiveness Reviews. Also known as OGB.
Quantitative – quantitative
is concerned with data analysis, opposite to qualitative.
Stakeholders – everyone
that is involved in a project so Oxfam, partner organizations, donors,
“Written by Emily Boyce who
is interning at Oxfam Novib’s impact measurement unit as a quantitative
researcher. All the way from England to explore how Oxfam Novib does things and
to study at the Institute of Social Studies, she does not shy away from a
challenge. She founded a toilet twinning campaign when studying for her
Bachelor’s degree at Warwick University and has now set her sights on new