What Really Goes Wrong in Evaluating Aid Projects?

Credit: Daily Nation, Kenopalo.com

In the first years I worked in the aid sector, I was part of a team implementing development projects. I was always excited when evaluation time came around, whether it was mid-term or final. I couldn’t wait to see how outsiders viewed our project, our work, and how we could improve. I learned that evaluation is an important phase in the project cycle, aimed at improving the effectiveness of the aid or assistance we deliver.

However, I was also nervous. Would the lack of progress, inaccurate design and approach, poor implementation, and weak monitoring of our project be exposed in the report?

My curiosity kept me observing closely. In several evaluations, I noticed that some findings I believed should be in the report disappeared, especially the sensitive or negative ones. To me, these evaluations went wrong. Decades later in my career, I found that truly thorough and honest evaluation reports are very rare.

These experiences made me question: How can evaluations go wrong despite the many technical guides available? And what are the consequences for the people these projects are supposed to help?

The OECD’s Six Criteria for Evaluating Aid

The OECD provides six criteria to guide the evaluation of development aid or assistance, specifically official development assistance (ODA): relevance, effectiveness, efficiency, impact, sustainability, and coherence. Using these criteria, evaluators should be able to determine if a project addresses the targeted issues, achieves its goals, uses resources efficiently, positively impacts beneficiaries, offers long-term benefits, and aligns with the overall development landscape.

Credit: OECD

While these criteria provide a structured framework, their practical application can face significant challenges. Those criteria often need to be tailored to the unique context of each project, as outlined in the Terms of Reference (ToR). Typically, a commissioning body — which could be the implementing agency, the donor, or another organization the donor has designated — creates the ToR.

The ToR typically covers technical aspects tailored to each evaluation before hiring a consultant team. These technical aspects include:

• Evaluation type and scope: whether it’s midterm, final, or both.
• Evaluation criteria: specific criteria tailored to the project’s context (mostly based on OECD criteria)
• Duration and working days to be allocated to do the evaluation:
  Data collection and methodology
• Skills and expertise of evaluators: ensuring the team has the necessary expertise.
• Utilization of evaluation findings: How the findings will be used, such as for project extension, suspension, cancellation (in a mid-term evaluation), planning a new phase, securing new funding (in a final evaluation), or deriving lessons learned for similar projects.

A solid but adaptable TOR usually forms the foundation of a successful, truthful, and useful evaluation.

Non-Technical Issues Affecting Evaluations

Despite a structured framework and a tailored ToR, non-technical issues can derail evaluations.

Conflict of Interest

Conflict of interest can lead to biased evaluations, especially when evaluators have been involved in the project or have a stake in its outcome. Some examples include:

• Self-evaluation: When the evaluator was involved in the design or implementation of the project, either as a staff member or consultant, hired by the aid providers (donor or implementing agency) or recipient institutions.
• Internal staff evaluators: When the evaluator is a staff member of the commissioning agency, either the donor or the implementing agency,
• Commissioning and implementing agency overlap: When the commissioning agency is also the implementing agency for the project being evaluated.

Credit: Marketing91

Self-Preservation Disincentives

Self-preservation disincentives can affect each stakeholder in the evaluation process.

• The project team (implementing agency) may be concerned about securing funding for the next phase or new projects if the evaluation report reveals numerous flaws in the current project.
• Donors’ staff: If the project has too many critical, negative, or sensitive findings, it may have a negative impact on their monitoring efforts, as the staff managing the evaluation is usually also in charge of monitoring.
• Recipients (government institutions): Critical findings may indicate that they failed to manage the project, lowering their internal performance scores.
• Recipients and beneficiaries (communities and NGOs): They may fear that too many critical findings will jeopardize future projects.
• Evaluators: Evaluators might worry about future job opportunities. If they don’t follow what the commissioning agency wants in the evaluation report, they might not get hired again. If there are many opportunities from the recipients’ or project team’s sides, how can they get hired if they report critical findings?

“I am a duck bent on self-preservation.” Credit: img.photobucket.com

Let’s assume that the evaluator has integrity and is unaffected by non-technical issues. Even so, the evaluation may not go smoothly. The evaluator may still face resistance from other stakeholders if the evaluation report contains sensitive or critical findings, as non-technical issues may still be relevant to those stakeholders.

Assume you are an independent consultant hired to evaluate a project using the OECD’s six evaluation criteria. You discover inefficiency and ineffectiveness in the project: funds are distributed as planned, but achievement indicators are not met. You incorporate these findings and recommendations into the draft report.

The project team, donors, and recipients may disagree.

On the surface, their responses may appear to be technical inquiries, requesting additional data, or questioning the methodology. However, these challenges frequently only address critical, negative, or sensitive findings.

Underneath these official clarifications, the real issue may be non-technical, arising from a conflict of interest or self-preservation, either consciously or unconsciously.

Handling Pushbacks

Here are some strategies to deal with these challenges:

Before taking the job:

1. Understanding the characteristics of the commissioned agency: Research the agency’s history, priorities, and potential biases to prepare for the evaluation’s context.
2. Conducting a brief stakeholder mapping: Read the ToR and project documents (usually attached to the ToR) to identify key stakeholders and their interests.
3. Assessing potential non-technical issues: Perform a self-analysis to identify potential non-technical challenges and decide whether to take the job.
4. Checking for cultural and ethical issues: Ensure that the ToR includes considerations for cultural and ethical issues, including cultural context and protection of the confidentiality of interviewees or respondents. This can help prevent misunderstandings and ensure that the evaluation respects ethical standards and local contexts.

If you are unsure if the ToR aligns with your “integrity standard,” don’t take or apply for the job.

Credit: Cartoonmovement

If you decide to take the job, be prepared for potential challenges when you report critical or sensitive findings.

Here are strategies you can apply.

During the report writing:

1. The inception report: The inception report is critical. Use it to outline plans for methodology and approach, including compliance with ethical standards like the General Data Protection Regulation (GDPR). Clearly state how cultural sensitivities will be handled.
2. Writing reports on sensitive or critical findings: Start with straightforward language for critical findings, anticipating pushback. This allows room for negotiation while maintaining essential findings.If you start with softened language, it might be altered or disappear at the end of negotiations.
3. Handling pushbacks professionally: Understand stakeholder behavior as part of non-technical issues; don’t take their comments personally. Use rational arguments to defend your findings and frame them as lessons learned or opportunities for improvement.
4. Strategic use of rationales: Save the strongest rationales and arguments for sensitive findings for later stages of discussion. This can provide leverage during negotiations.

Last Steps to Maintain Integrity

As a consultant conducting evaluations, we should strike a balance between (a) satisfying the clients (the commissioning agency) to keep our jobs and (b) maintaining the objectivity and integrity of our evaluations.

So, if none of the strategies help, should the evaluator compromise their work to keep their job? If yes, how far should they go?

I will refer to Koch’s book “Foreign Aid and Its Unintended Consequences” to answer this.

Even if a donor provides aid with good intentions, it can sometimes end up hurting the very people it’s supposed to help.

In a hypothetical situation, an inaccurate or misleading evaluation report can lead to or exaggerate the potential negative unintended consequences of a project.

Thus, personally, I will prevent any potential harm or negative unintended consequences for the beneficiaries or recipients.

If sensitive or critical findings and relevant recommendations that are rejected will potentially harm the beneficiaries or recipients, I will consider the following last steps:

1. Considering Withdrawal: If necessary, consider pulling out from the evaluation to maintain professional integrity, even if it means forfeiting the remaining fee.
2. Requesting Removal of Your Name: If the client insists on altering the findings, request to have your name removed from the report to protect your professional reputation. This may affect future opportunities with the same agency, but it maintains your integrity.

We may encourage donors and funders to recognize and address both technical and non-technical issues, adopting comprehensive evaluation practices that prioritize ethical considerations and integrity. However, those ideal conditions may be beyond our control.

What we can do as evaluators is prevent mistakes that may be harmful to the recipients and beneficiaries.

If that does not work, our last resort is to exclude ourselves from the mistakes by withdrawing or requesting the removal of our name from the report.