The Role of Evaluation

Generally, evaluation is the process of examining the effects of a spending policy or programme while it is in operation or after it has come to an end. The first step in undertaking an evaluation is to determine the main questions that need answered, for example, do we need to know if the policy was implemented as planned or do we want to know if it has achieved its objectives?

Evaluation should be seen as part of an overall policy making or decision making cycle (see figure 1.1) beginning with the RATIONALE for a particular spending programme. This means identifying either an area of market failure, the correction of which would improve the allocation of resources and/or identifying a need for a programme that will bring about a more equitable allocation of resources or meet a social need. Factors that bring about the failure of markets and other reasons for government intervention are explained in Section A of the technical supplement. The next step in the cycle is APPRAISAL where the costs and benefits of the options to address the particular problem are considered. Following the selection of the best option a programme is implemented and its performance in addressing the rationale of the programme is MONITORED. Monitoring will indicate the extent to which the programme is meeting its objectives. EVALUATION can then assess the full effects of a programme and recommend if change is necessary. This in turn leads back to appraisal of the options for change.

The questions asked will determine the most appropriate type of evaluation. Process and outcome evaluations are the most commonly used types of evaluation and differ in emphasis on the questions they set out to answer. A process (or formative) evaluation asks how can the policy be improved and made more effective and/or efficient. It involves considering:- how the policy has been implemented; how well it is moving toward the achievement of its goals; and what activities might improve it. The evaluation can reveal if external factors are interrupting the policy or programme or if the programme is operating appropriately or if the programme is being carried out as intended. This type of evaluation is useful where there is little understanding of how final outcomes are to be achieved. Process evaluations are particularly valuable in the early stages of a policy when there is likely to be a great deal of development and change.

Figure 1.1 The Role of Evaluation in the Policy-Making Process

An outcome (or summative) evaluation asks what can be concluded about the effects of the policy or programme. It is more concerned with the overall effectiveness and final impact of the policy. It is suited for a programme that is well established with clear and measurable goals. The key question for the outcome approach is whether to continue, expand, reduce or terminate a programme. However, regardless of the type of evaluation, the prime motive of evaluation is to identify lessons which can be applied to improve policy in the future.

Programme evaluation may appear to be a novel and complex exercise but much of the information required should be available from existing sources such as departments' Management Information Systems, Management Frameworks and associated performance indicators. This, taken with ongoing programme monitoring data, should provide much of the information required. However, it should be noted that these sources may not provide all the necessary information to undertake a comprehensive evaluation. Before initiating a programme therefore, thought needs to be given to the indicators needed for proper evaluation in order that the correct data can be collected by Departments. In short, adequate planning for evaluation should be an integral part of the policy making process.

Evaluation studies can be carried out at the policy, programme or project level. Whilst the distinction is not always clear cut, it is possible to apply broad definitions to each. Essentially the three can be seen as a hierarchy. A policy generally signals broad intent whilst a programme is more often concerned with specific action plans to achieve the policy, (usually there are a number of programmes in place to support the policy). The third tier is project evaluation. This usually deals with a discrete one-off scheme, such as building a community centre or assisting a company to invest in Northern Ireland. Again a number of projects are normally in place to support a programme, for example, many companies are given Selective Financial Assistance (SFA) by IDB (each is a project) whilst collectively they form part of the industrial development programme.

Policy and programme evaluations are concerned with assessing the effectiveness of the activity in achieving its objective and deciding whether this was done in the most efficient and economical way possible. Project evaluation is also concerned with assessing whether or not the objectives were met, though more emphasis is placed on assessing the accuracy of the forecasted costs and benefits in the project appraisal.

While the core conventions outlined in the guide provide a general framework for evaluation, it should be recognised that there is no one approach that can be applied to all evaluations or that will solve all evaluation problems. Evaluations may pose a range of questions (for example, how does the programme work? are the goals of the programme being met?) They may be required to serve different purposes (for example, is the programme cost-effective? would an alternative programme be better? should the programme continue?) It may be necessary to use different evaluation strategies at various stages in the life of the programme (for example, at the planning stage or when the programme is in operation). It is important to know which type of evaluation best answers the questions set and what sorts of questions a particular evaluation approach will not answer well. The aim is to design the most appropriate approach and tailor the method to answer the questions set for each particular evaluation.

Clearly, given the range of problems and constraints that might emerge, it is recommended that details of the evaluation approach are considered before the policy or programme begins. It is also important for evaluators and administrators to get together at every stage of the evaluation process. Generally, policy evaluations require specially designed research which will depend, amongst other things, on the programme to be evaluated and the techniques of data collection that are possible. The results from an evaluation design which falls short of the ideal (shortcomings should be acknowledged) can still help with decisions on the impact and future operation of the policy.