Read: organisations 4: Evaluating and monitoring

Managers, project leaders and funders or subsidizers want to know whether their investments and efforts yield the desired results, and if the work is done appropriately and efficiently. Therefore, efforts and results should be made visible and measurable.

Some projects have aims that can be measured accurately. For instance, 30% of the people in the political bodies chosen are women. With other projects, more indicators are necessary in order to measure whether their aims have been achieved.

For example: public safety. An indicator can be: the number of reported criminal offences has fallen by 20% within a year. This, however, could also be caused by the fact that women find it increasingly scary to walk in the dark on their own, and therefore decide to stay at home. So, more indicators are needed to measure public safety. For example: investigate what areas in a certain town women experience as unsafe, and improve these areas (lights, fewer bushes, more police patrolling). An additional indicator will be the extent to which women feel safer after a period of time. This can be investigated by interviewing local women from time to time.

There are also projects and policies which do not clearly formulate their ultimate goal, but rather a direction. In these cases it is not possible to formulate the ultimate goal. A project, for instance, in which different organisations co-operate (school, doctor, police) aims to detect situations in which children are not doing well at an early stage. All manner of things could be the matter with a child: bad school results, illness, or maybe it is always alone or on the street. It depends on the situation whether the project's partners decide to work together or take action separately. In other words, the partners' aims are not formulated in detail, but the direction of their co-operation is clearly defined. It would be possible, though, to look, after some time, at what kind of problems they come across, and what kinds of co-operation appear to work well. These two aims, looking at the kind of problems that occur and how the partners co-operate, can be evaluated.

Different projects are concerned with different things. When different parties work together, the parties will have both shared and individual interests; in this case different evaluations will be desirable. For example: three organisations co-operate in order to set up a public centre for women. In addition to this shared concern, each organisation has its own interests: survival through co-operation, establishing the surplus value of multi-cultural co-operation; expanding its own sphere of influence.

In some cases, those directly involved in a project and its financiers will want different kinds of evaluation. For example: a local council wishes to encourage more girls to choose technical careers. At the same time, it wants to loosen its involvement in the execution of such kind of policy plans. Therefore, a contract is drawn up with a number of schools and youth organisations in which the latter receive money to achieve the council's aims. The council does not become involved in how they do this; it is only interested in the result, not in the method applied. To the schools and youth centres, the way in which the aims are achieved will be an important point of evaluation as well. Different situations will require different methods of evaluation.

Below, we will explain a few concepts; then we will discuss the evaluation methods.

We need to make all kinds of choices with regard to the method of evaluation. A few concepts:

• evaluating means collecting data about the process or product and attributing a certain value to these data.
• the moment of evaluation: ex-ante evaluation: before a project or activity begins; formative evaluation: during the project; ex-post or summative evaluation: after the project has finished.
• the basis for the evaluation: evaluation on the basis of a plan, or the process, or the product.
• an assessment is made of the input, the output or the outcome. The input is what is invested, the output is the immediate result (for instance the number of participants at a conference); the outcome, or impact, refers to the continued effect (for instance the way in which the participants at a conference will use the information at a later stage).
• a one-off or permanent evaluation. By monitoring we mean permanent control. Monitoring can relate to a project with definite boundaries, or developments surrounding the project or organisation. (An example based on a health information project: the information itself can be measured continually, but it is also possible to monitor how certain developments in society have an influence on those who receive the information).

Evaluation methods that are used often:

1. the expenses-profits analysis: all the efforts and results are converted into money and balanced against each other. For example: this festival costs x, and will yield y in admission fees.
2. the costs-effects analysis: the expenses of a certain approach are expressed in money, and the profits are expressed both in money and in other effects. For instance: the percentage of girls that register for technical subjects, or the number of complaints coming in about unsafe streets. The costs/effects balance of one approach is then compared with the costs/effects balance of other approaches.
3. the goals achievement matrix: the measures are tested on the basis of the contribution they make to larger goals. For example: this measure's effect on safety is x, while another measure has y effect.
4. the multi-criteria analysis: the evaluation is based on a combination of criteria. For example: increased safety is measured on the basis of fewer complaints from women, an increased sense of safety, and a larger number of women who go out.
5. effect reports: before a project begins, the project's or policy's effect on, for instance, the environment or the position of women is measured.

New developments

These methods are used fairly regularly. It is characteristic, that while many projects are tested for certain criteria when they begin, for the duration of the projects, or even afterwards, hardly any efforts are made to find out whether they have achieved the required results. Today other evaluation methods, which are linked more closely to modern developments and demands, are used increasingly. A few important developments include:

• the need for permanent forms of evaluation, rather than a one-off evaluation beforehand or afterwards, has increased; those involved want to know about progress and results while the project is going on as well. The most important advantage of this is the option to learn from mistakes. It is possible to react, and maybe adjust, at an earlier stage.
• the relations between financing and receiving party are changing in many places. The subsidized party is no longer an object that only needs to be evaluated and given direction. We now see a shared interest and emancipated 'receivers' who take part in the evaluation themselves as well.
• another party, that of the policy's or project's target group, is emancipating as well. A customer-oriented perspective and the recognition of the necessity of having policy with a broad basis will lead to a more participatory approach of projects/policies and their evaluation.
• after all the emphasis there has been on quantitative data, there is now a move towards more interest in qualitative data.
• there is more recognition now for the multiformity of interests and goals of the different parties. The equality of parties means that in some projects different parties can pursue their own secondary goals, and that the existence of different goals in one project can and should be clarified to all the parties involved.

Evaluation methods which fit in with these developments include:

• market research with the help of citizen panels or focus groups; it is investigated whether the customers or target groups need a certain measure or under what circumstances they could make use of a specific service. In this kind of investigation the interests and needs of the customers take a central place instead of, or in addition to, for instance, the government's goals.
• the realization test: it is investigated beforehand whether a certain (policy) plan can actually be realized. This is done, for instance, by investigating whether the plan would find a lot of support among its future users.
• the so-called fourth generation evaluation method: evaluation aims are determined together with all those who are involved; the evaluation itself, which is not a one-off occasion but a continual process, is also carried out with all those who are involved. The evaluator will act as the mediator between the different parties. He/she will negotiate about claims, concerns and issues (claims: what all the parties agree should be the results; concerns: business that still needs to be worked out; issues: points about which the different parties disagree because they mean a profit for one party and a loss for the other).

Indicators

With all these evaluation methods it is necessary to determine indicators in order to measure the results. With the more modern evaluation methods, the setting up of these indicators is done by (the representatives of) all those involved. A good indicator will comply with the following criteria:

• the availability of data: is it possible to collect the intended data? (eg data about happiness is much more difficult to collect than data about income and health).
• can the indicator be influenced? (eg you can hardly influence the weather) and: can the parties involved affect it? (eg some persons/organisations have an influence on the quality of drinking water, while others have not).
• can it be compared with a certain norm? (eg the position of women in other countries, or during the last decade, or in other organisations).
• does the indicator say anything about the broad spectrum of input, process, output and outcome?

To sum up, evaluating is a participatory, on-going, educational and dynamic process.