Economic Appraisal Guidance
DFP Corporate Logo Economic Appraisal Guidance

NI Practical Guide to the Green Book

APPENDIX 5

APPRAISING AND EVALUATING NEW TARGETING SOCIAL NEED
   

Introduction
A5.1   This appendix provides guidance on how to take account of New Targeting Social Need (New TSN) objectives in appraisals and evaluations of policies, programmes and projects. It incorporates and replaces the guidance issued by DFP in November 2001.
A5.2   Appraisals in which New TSN is relevant should cover all of the standard appraisal steps in accordance with the general guidance in the NI Practical Guide. However, consideration of New TSN is particularly important at the steps 'establish the need for the project', 'define objectives and constraints' and 'weigh up non-monetary costs and benefits'. The remainder of this note focuses on how to consider New TSN at these stages.
   

Consider the relevance of New TSN early in the appraisal process
A5.3   The first consideration is to determine whether or not New TSN is relevant to the policy, programme or project being appraised. This should be established at the outset of the appraisal process, when defining the needs and objectives to be addressed, so that relevant New TSN needs and objectives can be taken into account from the beginning.
A5.4   Appraisers need to be aware of the importance of thinking about New TSN at the commencement of appraisals. The New TSN dimension is more likely to be appraised successfully if it is identified and built into the appraisal at the outset, rather than at a later stage. For instance, the New TSN impact will be greater if New TSN criteria are used to influence option identification, than if consideration of New TSN impact only begins after the options have been determined.
   

Defining New TSN-relevant Needs
A5.5   Early in any appraisal, it is important to establish the need for expenditure by, for example, identifying deficiencies in current provision and specifying the projected need or demand for services. For instance, where New TSN is relevant, it will be appropriate to consider and identify the specific people, groups or areas of social need who are to be targeted. The poorest should be identified objectively using measures of deprivation, fairly and consistently applied within programmes.
A5.6   The method of targeting will depend on a number of factors, notably the level at which the programme is delivered. For instance, where programmes are delivered to individuals, Departments should identify the people who are least well off and who need the particular service most. One model that has proved successful is to undertake proactive initiatives, often in partnership with others including the voluntary sector, designed to improve the take-up of income-related benefits. These involve identifying and targeting particular groups and individuals and include action to remove or reduce barriers such as difficulties with language, poor literacy, little knowledge of entitlement or a fear of claiming.
A5.7   Some areas and communities are subject to higher than average rates of unemployment and are more deprived than others. New TSN recognises this and commits Departments to target such geographic areas for special attention, where possible. Programmes such as the proposed Neighbourhood Renewal Strategy, the Rural Development Programme and other area-based interventions are particularly relevant in this regard. Indicators of multiple deprivation are an effective means of identifying the most disadvantaged areas when programmes attempt to address a range of different needs.
A5.8   NISRA has published a report titled Targeting Areas of Social Need in Northern Ireland (NISRA Occasional Paper No.16, April 2001), which specifically addresses this question. It reviews various approaches that have been used to measure deprivation, with particular emphasis on the 'Robson measures'. These served as a basic tool for identifying the spatial distribution of disadvantage in Northern Ireland since 1994, but they have been superseded by the 'Noble measures' - as is explained below.
A5.9   An illustration of the use of measures of deprivation is the way that they have contributed to the classification of areas used for targeting. For example:
  • the International Fund for Ireland's (IFI) set of designated disadvantaged wards
  • the Community Economic Regeneration Scheme (CERS)
  • the Community Regeneration and Improvement Special Programme (CRISP)
  • the Making Belfast Work and Londonderry Initiatives
  • the Peace and Reconciliation Programme
  • the Building Sustainable Prosperity Programme
  • the Rural Development Programme
  • the LEADER II Programme
  • the designation of TSN areas at Local Government District level for use by IDB, IRTU and NITB
  • The designation of TSN areas at Ward level for use by LEDU and T&EA
A5.10   Measures of deprivation will have a continuing role in the targeting of need. NISRA commissioned research to review deprivation measures in Northern Ireland, conducted by a team led by Mike Noble of the University of Oxford, including the Queens University of Belfast. The results of their efforts have been published in the report Measures of Deprivation in Northern Ireland (Social Disadvantage Research Centre, Department of Social Policy and Social Work, University of Oxford, June 2001) which is available from NISRA.
A5.11   The Noble Report contains a comprehensive new set of deprivation indicators, based largely on 1999 data. These are known as the new Northern Ireland Measures of Deprivation. They include:
  1. 7 Domain Deprivation Measures - covering Income; Employment; Health; Education, Skills & Training; Geographical Access to Services; Housing Stress; and Social Environment
  2. a Multiple Deprivation Measure, based on a combination of all the Domain Deprivation Measures
  3. a Child Poverty Measure, which is a subset of the Income Domain Deprivation Measure
  4. an Economic Deprivation Measure, which combines the Income and Employment Domain Deprivation Measures
A5.12   Results have been produced at various levels e.g.
  • at ward level for (i), (ii) and (iii) above;
  • at enumeration district level for (iv) above, and for the Income and Employment domains within (i) above; and
  • at Local Government District level for (ii) above
A5.13   These results should provide a useful tool to assist with targeting area-based need. The choice of measures or indicators is important if the extent of need is to be gauged accurately. For example, incidence of Free School Meals might act as a useful proxy for income deprivation in some circumstances. However, such an indicator is generally unlikely to be as appropriate as a measure based directly upon Income data.
A5.14   In addition, the spatial range for the measure or indicator needs to be carefully selected to draw out precisely where the need lies. For instance, a measure at Ward level might suggest limited deprivation but it could be masking a few very deprived areas within that Ward - as a result it may be more appropriate to use a measure based upon smaller areas, for instance, Enumeration Districts, in order to unearth the real pockets of need. As far as possible, the geographic range of the selected measure or indicator should match that of the project or programme in view, otherwise there is potential for poor targeting of resources.
A5.15   The new Northern Ireland Measures of Deprivation are important, but other information sources also have a vital role to play. There is a wealth of social and economic statistics on Northern Ireland to draw on. For instance, the labour market statistics produced by the Department of Enterprise, Trade and Investment (DETI) will be relevant to the targeting of measures to increase employment and employability. Other Departments have their own data and information sources that should be used to help target their programmes.
A5.16   Another consideration in targeting resources on an area basis is that the areas to which one type of programme is appropriate, for example, urban regeneration, may not be appropriate to other types of programme, such as rural development or forestry. The need for the particular service must also be taken into account. While areas may have similar overall levels of deprivation, their need for a particular service may differ.
A5.17   There is no unique method for identifying needs. Different programmes have different objectives and target groups, and targeting has to be tailored to suit the specific programmes in view. Departments must judge the approach that is appropriate to each circumstance, drawing on specialist advice as necessary.
   

Define New TSN-relevant Objectives
A5.18   Having identified the specific people, groups or areas to be targeted, it is good appraisal practice to set precise and measurable objectives and targets. These should serve as parameters for the identification of options, and, importantly, provide a basis for appraising the comparative merits of options, and later evaluating post-implementation achievements.
A5.19   New TSN has a particular emphasis on tackling the problems of unemployment and increasing employability. Poverty tends to be associated with other types of social need, and New TSN focuses on addressing the inequalities and needs of disadvantaged people in areas such as health, education and housing. Accordingly, within a particular appraisal, it may be appropriate to specify objectives directed towards achieving:-
  • increases in employment opportunities for unemployed people;
  • increases in employability i.e. by facilitating the development of aptitudes or skills people need to equip them to work or reducing barriers which prevent them from working; and
  • improvements in the position of the most disadvantaged people by:-
    • reducing inequalities in health, housing, and other relevant areas which are related to disadvantage
    • reducing inequalities in access to or take up of services where these are related to social need.
A5.20   These provide qualitative indications of New TSN-relevant objectives. However, when defining objectives in live cases, they should be expressed in terms that are SMART i.e. Specific, Measurable, Achievable, Relevant and Time-dependent.
A5.21   In particular, the importance of including measurable objectives must be stressed. Without them, there will be little basis for appraising and evaluating the comparative performance of options. Accordingly, objectives need to be expressed in quantitative terms as far as possible.
A5.22   The New TSN Action Plans provide specific examples of New TSN-related objectives and targets. These may be of direct relevance to some appraisals and evaluations, or they may be adaptable to suit them.
A5.23   Where there are multiple objectives, there may be merit in prioritising them. It may be the case that the policy, programme or project in view will address both New TSN-relevant and New TSN-irrelevant needs and objectives, in which event it may be appropriate to prioritise the New TSN and non-New TSN objectives. Even if all the objectives are New TSN-related, prioritisation of the objectives may help to focus the appraisal.
   

Assess the New TSN Impact of the Options
A5.24   This is about analysing the extent to which the options will achieve the New TSN objectives specified in the appraisal. It requires consideration of the costs and benefits arising from the proposal. The costs of any measures needed to deliver the policy, programme or project should be estimated in accordance with general appraisal costing guidance. However, New TSN-specific benefits are likely to be more difficult to capture in monetary terms, and their assessment will normally be addressed under the general appraisal heading of 'weigh up non-monetary factors'.
A5.25   The method for assessing New TSN impact should be decided on a case-by-case basis, taking account of the circumstances. There is no unique way of doing this. Often the first step will be to identify that the benefits of a particular option will impact upon a designated New TSN area. In some cases, for example, when determining eligibility for a particular rate of grant, this may be all the impact analysis that is required. However, it may often be appropriate to go beyond this and attempt to assess the nature and extent of the New TSN impact.
A5.26   A simple but effective approach is to develop an impact statement. In essence, this consists of a table summarising the impact of each option upon the project objectives, including each relevant New TSN objective. The size of the table can be scaled to suit the needs of the case in hand. An accompanying commentary summarising the main trade-offs between the options is generally helpful.

Impact Statement

  Option 1 Option 2 Option 3

New TSN Objective 1

      

New TSN Objective 2

      

New TSN Objective 3

      

Other Objective 1

      

Other Objective 2

      

Other Objective 3

      
A5.27   The cells of the table should contain suitable quantitative impact measures or indicators; and/or qualitative impact analysis. For example, suitable economic and social indicators may be used to indicate the New TSN benefits, and thus help to compare how the options perform against the appraisal objectives, and how they compare with each other.
A5.28   A few broad examples of suitable indicators are given below for illustration. These will not always be relevant, and the list is far from definitive. There is a need to choose carefully, drawing on sources such as the Noble Report and the wide range of available Northern Ireland social and economic statistics, to come up with the right indicators to suit the individual policies, programmes or projects in view. Statisticians and economists can help with this.

Examples of Relevant Economic and Social Indicators

Employment

   jobs promoted or created
placements of long term unemployed or disabled

Employability

   trainee places provided
% of population in training/further education

Housing

   new starts and rehabilitations
rates of homelessness and houses lacking amenities

Health

   cases treated
rates of accidents, incidence and death

Education

   enrolments and class sizes
% of pupils achieving grades/qualifications
A5.29   It may be appropriate to supplement quantitative analysis based on indicators like these with qualitative assessment, including textual description of impacts that are difficult to quantify, and interpretation of the quantitative material. For example, figures on jobs created can be supplemented by description of the quality of the jobs and interpretation of how beneficial they will be to the targeted people, groups or areas.
A5.30   When assessing the New TSN impact, consideration should be given to the deadweight and displacement associated with each option. Generally speaking those options that have the lowest incidence of deadweight and displacement will target resources best.
A5.31   Deadweight is expenditure targeted on a desired activity that would have occurred anyway, or on people or areas that do not need it. For example if it were established that the cost of public transport was a barrier to access to services in a particular New TSN area, then an option providing free travel to all would carry proportionately more deadweight than, say, an option restricting free travel to people in receipt of means tested benefits. Analysing the scope for deadweight inherent in each option can help develop proposals that target resources better.
A5.32   Displacement is the degree to which an activity promoted by a programme is offset by reductions in other similar activities. For example, granting assistance to a business in a New TSN area may displace the trade of similar businesses already operating there, or in adjacent areas. Alternatively, a proposal for a community facility may duplicate some other existing or planned services in the area or elsewhere nearby. Minimising displacement helps to maximize the New TSN impact.
   

Balancing New TSN Impacts with Other Factors
A5.33   Consideration of New TSN objectives and impacts must be balanced by consideration of other relevant objectives and impacts. For example, achievement of value for money (VFM) is a general objective of policies, programmes and projects. An option offering additional New TSN benefits may offer less VFM in terms of cost per output than an alternative option. In this example, there would be a need for judgement of whether the extra cost is worth bearing to obtain the extra New TSN benefits.
A5.34   In any given case there may be a variety of other objectives or constraints to be considered, for example operational requirements, policy objectives such as transport, health, education etc, and other 'difficult to value' factors as referred to in Section 2.7 of the NI Practical Guide. The relative priority of New TSN vis-à-vis these other considerations will vary according to the circumstances and should be judged and documented case by case
A5.35   In some cases the primary objectives of a proposal may reflect New TSN concerns. In others, the primary motive may be something else entirely, but it may still be appropriate to assess New TSN impacts. On some occasions, New TSN may be of no relevance. In any event, appraisal reports should provide clear analysis of all the relevant considerations, in order to facilitate the necessary judgements.
A5.36   It is important that appraisals and evaluations identify all of the relevant objectives, impacts and unquantified considerations, and present them in such a way that decision makers are informed of the trade-offs between New TSN, VFM and the other relevant factors. The approaches referred to in Section 2.7 of The NI Practical Guide can be employed for this purpose. For instance, as indicated at para A5.26 above, an impact statement may be used to tabulate the effect of each option not only upon New TSN, but also upon each of the other relevant objectives, constraints and other factors.
A5.37   It is good practice is to cover all non-monetary factors by either the impact statement method or the weighted scoring method. It is not helpful to cover some factors in a weighted scoring calculation and others in an impact statement. This can cause confusion and invalidate the rankings emerging from the weighted scores. For example, it is not generally helpful to treat New TSN or Equality separately from a weighted scoring of other factors.
   

New TSN and Ex-Post Evaluation
A5.38   The Office of the First Minister and Deputy First Minister (OFMDFM) is responsible for evaluating the overall progress of New TSN within a framework that encompasses the Actions Plans of all NI Departments. However, this does not avoid the requirement for Departments themselves to consider how to allow for New TSN within their regular programmes of evaluation. The following are some basic principles for doing so.
A5.39   General guidance on evaluation is given in Section 11 of the NI Practical Guide. It requires that: "Every appraisal of any substance should indicate how the proposals concerned will be evaluated after completion and how the results of the evaluation will be disseminated. This applies to policies and programmes as much as to projects". This is not a new requirement, thus it applies to proposals with a New TSN dimension that are already under way, as well as to new cases.
A5.40   Ex-post evaluation examines the outturn of a policy, programme or project and seeks to learn lessons from experience that will help in future planning and implementation. It is important to establish a baseline option or 'counterfactual', against which to compare the actual outcome, in order to estimate net additionality, that is, the benefits (including New TSN benefits) that are attributable to the policy, programme or project.
A5.41   Evaluation is like appraisal conducted in retrospect, hence the principles outlined above for appraisal of New TSN will also apply broadly to its evaluation. The New TSN objectives and indicators employed at appraisal stage will generally be equally relevant when it comes to evaluating success post implementation. For example, if an appraisal projects job creation in a New TSN area, it would be appropriate subsequently to evaluate the jobs actually created, and specifically how they impacted upon the population in the New TSN area.
A5.42   Meaningful evaluation of New TSN activities depends upon suitable preparation at the appraisal stage, particularly by recording good quality baseline information and establishing the baseline option. It is generally helpful to include details of the pre-implementation deployment of resources, levels of service provision, and social and economic indicators. Objectives, key calculations, and assumptions should be recorded in detail. Systems for recording relevant information during and after implementation need to be established. This all helps to aid comparison of outturns with estimates, assessment of success in achieving objectives, and consideration of how processes can be improved and resources better targeted to achieve New TSN aims.
   

Help and Advice
A5.43   Specialist advice may be needed. The New TSN Unit within OFMDFM's Equality Directorate can provide general advice and guidance on New TSN - see 12.4.8 for contact details. The OFMDFM Research Branch provides research and statistical support. Timely involvement of NISRA statisticians and Departmental economists can help to ensure that New TSN impacts are properly assessed and incorporated into appraisals and evaluations.