A Non-Expert at an Expert Meeting : Methodology, Measurement… Merlot?
* All starred words are explained in the ‘Glossary of Expertise’ at the end
International Women’s Day is celebrated around the world with dancing; singing; eating and celebrating gender equality achievements globally and making steps towards future parity. However, in the Oxfam Novib Impact Measurement* Team, the 8th March was instead a day for celebrating methodology*; measurement; evaluation and impact at Humanity House in the centre of Den Haag. As the new intern on the quantitative* side of the team I was invited to attend the day.
It began at 8.30 as the team gathered in the building, finding projectors and flipcharts and welcoming people as they ventured into the basement where the plenary sessions would take place. My Dutch vocabulary expanded as I directed people to the cloakroom:
“garderobe” and for tea and coffee “thee of koffie?”, although my incapacity to respond to any questions more complicated than “where are the toilets?” quickly gave me away as a non-Dutch imposter. I snuck into the back of the opening session and learnt about the necessity of failure for learning – should we have a mandatory level of failure*? Other themes that reoccurred throughout the day were questions such as: how do theory and practice relate? How to involve key stakeholders*? How to create ownership*
of evaluations? How to learn from results?
Photo credit: Tessa Moolenaar
At the back of the plenary session I was able to nod along and take notes as if I too was an expert but in the breakout sessions this changed. After hiding on the last row during the presentation on an initiative called Balika trying to reduce child marriage in Bangladesh (which was a mistake as I am only 158cm tall…) we split up into smaller groups where the real experts were able to critically reflect on the presentation and results. I realized very quickly how little I knew about development projects – I had no idea about the issue of child marriage (this project aimed at improving the skill level of girls so they were perceived as an asset and not a liability); I had no idea about what are considered “good” results (apparently a change of 20% in 18 months is a surprisingly high result) and I had no idea about the differences between, outputs; outcomes and impact (please see glossary). The learning curve was steep but the climb was aided by
the supply of biscuits, tea, coffee, and, of course, lunch.
Photo credit: Tessa Moolenaar
Lunch was one of the highlights of the day: it was an incredible feast of salads and sandwiches and little sweet treats. It was also an opportunity to meet with some of the ‘experts’, who were very willing to share their work and experiences. I met one woman who had spent a year in a Nigerian village growing her own food to exchange with
her neighbours (there was a roadblock by armed rebels, preventing any food from reaching the village) during her time working on a local finance initiative. Everyone could also “run like a refugee” in Humanity House’s immersive experience which I did, ending the experience by pulling a trigger on some of the experts who had “run” earlier – something I would not like to repeat in reality!
Photo credit: Tessa Moolenaar
The second breakout session simply confirmed what the morning had already taught me: that my expertise lies more in drinking tea than in impact measurement. However, the presentation from OxfamGB* really interested me. Their emphasis on investing in skills of the people in partner organisations and giving back to the local community caught
my attention. For example, the data is immediately made available to the community once everyone has been interviewed – allowing them to see, for
example, how many goats the community has, or what is the average perception of the occurrence of GBV*? I think that skills investment is key to creating sustainable change and giving back means the interview relationship is not abused by taking too much time and knowledge without returning anything.
The day ended with one final get-together, exchanging the last pieces of advice and conducting an online survey (it was an impact measurement event after all…) to see which pieces of advice we collectively liked the most. The final
part of the day is where my expertise came in useful – drinking red wine, eating
snacks and chatting to new people at a borrel*.
Glossary of Expertise
Borrel – Dutch word to describe drinks and chatting, normally accompanied by snacks such as meatballs and bitterballen. It is gezellig!
GBV – an acronym of Gender Based Violence; “Gender-based violence
involves men and women, in which the female is usually the target, and is
derived from unequal power relationships between men and women.” It is often
used interchangeably with the term “Violence against women”.
Impact – contested definition but most people consider it a long-term result of a project (both positive and negative) such as higher income or greater employment opportunities.
It is often derived using outputs and outcomes but is much harder to
Impact measurement – the measurement of the impact of various projects. Not to be confused with MEAL (or MEL) – monitoring, evaluating and learning which is similar, but often more short term.
Mandatory level of failure
– for example this might be a rule that states that 20% of all projects have to be failures; the motivation behind this is to allow honesty and greater learning.
Methodology – the process of how the research is carried out, so for instance it could be surveys and then followed up with interviews, or it could be participatory action research.
Outcome – the result of an output, so for instance an increase in the number of children attending schools.
Output– a countable change eg the number of schools built.
Ownership – the idea that project leaders/beneficiaries/stakeholders need to be involved in all aspects of the project so they feel the results are applicable to them and they can use them, so it is not just an external requirement.
OxfamGB – the biggest Oxfam affiliate with a different method of carrying out impact measurement to Oxfam Novib, these are known as Effectiveness Reviews. Also known as OGB.
Quantitative – quantitative is concerned with data analysis, opposite to qualitative.
Stakeholders – everyone that is involved in a project so Oxfam, partner organizations, donors, beneficiaries etc.
“Written by Emily Boyce who is interning at Oxfam Novib’s impact measurement unit as a quantitative researcher. All the way from England to explore how Oxfam Novib does things and to study at the Institute of Social Studies, she does not shy away from a challenge. She founded a toilet twinning campaign when studying for her Bachelor’s degree at Warwick University and has now set her sights on new heights!”