Economic Appraisal Guidance
DFP Corporate Logo Economic Appraisal Guidance

NI Practical Guide to the Green Book

11.  

Evaluation
11.1  

Introduction
11.1.1   Evaluation plays an important role complementary to appraisal. Evaluation is an ex post activity which examines the outturn of a project, programme or policy, and is designed to ensure that the lessons learned are fed back into the decision-making process. This ensures government action is continually refined to reflect what best achieves objectives and promotes the public interest.
11.1.2   Departments should make arrangements to measure outturns and record them. Outturns should be compared with initial estimates and the results used to consider how to improve the quality of the assumptions in future appraisals, including, for example, the estimates of costs and benefits and the assumptions made about risks and appraisal optimism.
11.1.3   Evaluation is like appraisal conducted in retrospect. Thus the general principles and techniques of the NI Practical Guide and the Green Book apply as much to evaluation as to appraisal. This includes the principle of proportionate effort.
11.1.4   When any policy, programme or project is completed or has advanced to a pre-determined degree, it should undergo a comprehensive evaluation. Major or on-going programmes, involving a series of smaller capital projects, must also be subject to ex post evaluations (also known as post project evaluations (PPEs) or post implementation reviews (PIRs). Major expenditures or changes in resource use should be followed by full scale post implementation reviews. Lesser decisions require a more modest evaluation effort.
11.1.5   Every appraisal of any substance should indicate how the proposals concerned will be evaluated after completion and how the results of the evaluation will be disseminated.
11.1.6   PPEs or PIRs are an integral part of the process involved in completing a project and should not be seen as an additional complication to the appraisal process, but rather as an opportunity to learn valuable lessons and to avoid repeating mistakes.
11.1.7   An evaluation might address a project, programme or policy, particular aspects of one of these activities, or key issues affecting a number of activities. Where a large number of small scale projects or activities requires evaluation, it may be appropriate to select a representative sample for evaluation. It may be appropriate to conduct more than one PPE for a particular project, for example, where it has been implemented in stages.
11.1.8   The proliferation of terms in this area can be confusing. For example, the PRINCE2 guidance envisages evaluation at two stages:
  1. A Project Evaluation Review (PER), sometimes known as a Project Management Evaluation (PME):
    • The Project Manager is responsible for this evaluation, which should be conducted at project closure i.e. on implementation. It should result in an End Project Report and a Lessons Learned Report as described in the PRINCE2 guidance. This is concerned with the effectiveness of the project management up to the point of project closure. While it is useful and should be undertaken, it is not a post implementation assessment and is not a substitute for a PIR or a PPE.
  2. A Post Project Review (PPR), sometimes known as a Project Benefit Evaluation (PBE):
    • This should be conducted 6 to 12 months after project closure by an individual outside the Project Board and Project Team. The PPR is essentially a PIR or PPE. Although the PRINCE2 guidance emphasises the evaluation of benefits achieved, it is also generally expected to examine outturn costs.
Evaluation Terms
PPE Post Project Evaluation
General term for an ex post assessment of a project
PIR Post Implementation Review
General term for an ex post assessment of a policy, programme or project
PER Project Evaluation Review
PRINCE2 term for an ex post assessment of management effectiveness, conducted at project closure i.e. on implementation
Also known as a Project Management Evaluation (PME)
PPR Post Project Review
PRINCE2 term for an ex post assessment of benefits obtained from an activity, conducted 6 to 12 months after implementation
Also known as a Project Benefit Evaluation (PBE)
11.2  

Conducting an Evaluation
11.2.1   Evaluations should be led by individuals who have not been involved in the management or implementation of the proposal under consideration. This is so that they are in a position to take an independent and unbiased view. Basic principles for planning and managing evaluations are outlined at 1.6 above.
11.2.2   It is important to begin planning for evaluation at the appraisal stage and ensure that appraisal reports contain the information needed for evaluation. This should include an outline plan, setting out the general boundaries of the proposed evaluation, as indicated at 2.9.12-15 above.
11.2.3   An evaluation, whether it is called a PIR, PPE, PPR, PBE or something else, should normally follow this sequence:
  1. Establish exactly what is to be evaluated and how the outturns can be measured.
  2. Define the counterfactual i.e. estimate what would have happened if the intervention (e.g. the project, programme policy or financial assistance) had not occurred.
  3. Compare the outturn with the target outturn, and with the effects of the chosen counterfactual(s).
  4. Present the results and recommendations.
  5. Disseminate and use the results and recommendations.
11.2.4   The status quo or other baseline option used in the original appraisal should normally inform the counterfactual. However, viewing events from a post hoc position, evaluators may judge that the counterfactual would actually have been quite different from what was envisaged at the time of the appraisal, due to, for example, alternative states of the world and/or alternative management decisions. In such circumstances it may be helpful to consider other counterfactuals in addition to the original baseline option. The streams of costs and benefits that would have occurred in the counterfactual(s) should be estimated and set out so that the actual outturn costs and benefits can be compared with them.
11.2.5   The above sequence applies broadly as much to projects as to policies and programmes. However, proportionate effort should be applied e.g. when dealing with individual projects, particularly smaller projects, there tends to be less emphasis upon detailed consideration of alternative counterfactual states of the world. In most cases effort should be concentrated upon evaluating the extent to which objectives have been achieved, whether assumptions have proved accurate (e.g. by comparing outturns with target outturns), and what lessons can be learned.
11.2.6   In general, evaluation reports should summarise
  1. whether, and if so, why the outturn differed from that foreseen in the appraisal;
  2. how effective the activity was in achieving its objectives, and why;
  3. the cost-effectiveness of the activity; and
  4. what the results imply for future management or policy decisions.
11.2.7   The results obtained should generally lead to recommendations for the future. These might include, for example, changes in procurement practice, improvements to methods for estimating costs or benefits, changes to management procedures, or the continuation, modification or replacement of a project, programme or policy.
11.2.8   The results and recommendations should feed into future decision making. The methods used to achieve this may require senior management endorsement. Efforts should be made to disseminate the results widely within the organisation, and for this purpose it may be useful to employ summaries of the main points, and synthesis reports incorporating the results from a number of evaluations with common features.
11.3  

DFP Monitoring of Post Project Evaluations (PPEs)
11.3.1   In January 1993 the Audit Review Group (ARG) examined the control of capital projects and recommended that DFP Supply should rigorously pursue and monitor the completion of PPEs. Supply Branches regularly monitor Departments' performance in completing PPEs with the assistance of Economic Appraisal Branch.
11.3.2   In the interest of ensuring good practice and VFM, DFP may occasionally require Departments to produce a list of all projects approved in the previous year which were below delegated limits, but above de minimis levels, and to specify when these projects will be, or have been, completed and whether PPEs have been undertaken or when they are due to commence. (De minimis levels will be notified in the forthcoming new DAO letter referred to at 9.2.3 above).
11.3.3   DFP may ask to see a sample of these PPEs from time to time by way of quality assurance. DFP may also ask for examples of documented client management procedures, again as a quality assurance measure.
11.3.4   All projects should be subject to proper monitoring and control measures including PPEs for expenditures above de minimis levels and, in the case of capital works, completion of appropriate project reports. Such measures ensure good VFM by identifying difficulties, preventing the repetition of mistakes and identifying positive points which may be of use in other projects and/or other departments. In future, DFP approval of all projects above delegated limits will be conditional upon satisfactory arrangements for PPEs.