Limitations of outcome and impact assessment

The question of a project’s effects is as important and legitimate as it is difficult to answer in any definitive, comprehensive and explicit way.

Methodological limitations

Even if it can be proved that an individual project or programme has an effect, it is particularly difficult for aid agencies with many different projects to make assertions about the effect of the entire organisation is having or to determine positive macroeconomic and social outcomes of development work. In an ideal world, effects would be clearly proved for the entire system with the involvement of stakeholders and using rigorous impact assessment methods. That entails the use of quantitative methods supplemented by qualitative methods so as to obtain a comprehensive explanation of the links and processes. In reality, though, it is virtually impossible to meet these high standards. In addition, budgets are usually tight. It is crucial that every organisation sets priorities and fulfils the main demands with a mix of methods that is adapted to its circumstances.

Financial limitations

There is growing concern that reporting duties that are increasingly based on quantitative impact and outcome indicators will become so expensive that too many resources will be used up on project monitoring rather than for the target groups, and that they are hindering the flexibility that is necessary in participatory development projects. It makes sense to budget between 0.5 and 2.5% of the total annual cost of projects and services on project monitoring for impact assessment. In years when actual expenses are lower, the unused funds can be allocated to an impact assessment fund held in reserve.

This allows specific impact assessment projects to be carried out on a regular basis, e.g. every four years. Organisations with many small projects must choose which projects they want to assess. The following criteria can be applied to make this choice:

  • The scale of the project;
  • The size of expectations regarding the impact;
  • The degree of political significance;

As a general rule, more resources should be set aside for impact assessment in especially innovative projects in which little is known about their effects and results chains than for routine projects.

Normative limitations

An impact assessment system that is adapted to the organisation’s individual circumstances helps to find answers to important issues regarding its activities. Impact assessment should not, however, assume such importance that aid agencies subordinate their entire decision-making to impact assessment findings and only do things that are measurable and present good findings.