Details

Category: Policy, Projects & Programmes

Published: Tuesday, 27 May 2014 11:38

Written by: Matt Pye - HWD

Overview

Day One

Policy Evaluations – What Are They and Why Do We Need Them

• What do participants understand by policy evaluation?
• What is evaluation and why is it essential?
• Policy evaluation and evidence-based policy
• What factors affect how a policy should be evaluated?
• Evaluation as part of the policy cycle
• The Magenta Book

Types of Evidence - Establishing a Theory of Change

• Activities, Outputs, Outcomes
• Sequencing a logic model – building a causal chain
• Data required for a theory of change
• Case Study

Types of Evidence – Systematic Reviews and Meta-Analysis

• Harnessing what is already known
• Different types of research synthesis
• The structure of a systematic review
• Examples of Statistical Meta-Analysis for Policy Making
• Sources of sound evidence from systematic reviews

Types of Evidence for Policy Evaluation – Surveys, Census Data and Administrative Data

• Using existing survey data
• Collecting survey data Using census data
• The use of administrative data
• Risk of bias with surveys, census data and administrative data

Types of Evidence for Policy Evaluation – Qualitative Methods and Data

• In-Depth interviews
• Group interviews and focus groups
• Other consultative methods
• Observational and Participant –Observation methods
• Ethnography
• Case Studies
• Quality of Qualitative Evaluation
• The contribution of qualitative data to policy making

Establishing a Strong Counterfactual

• The importance of the counterfactual for policy making
• Identifying policy options
• Establishing the net effect of a policy

Types of Evidence for Policy Evaluation – Interrupted Time Series Analysis

• The logic of Interrupted Time Series Analysis
• Examples of Interrupted Time Series Analysis
• Risk of bias with Interrupted Time Series Analysis
• Examples of Interrupted Time Series Analysis in policy evaluation

Day Two

Types of Evidence for Policy Evaluation – Single Group Before and After Designs

• The logic of a single group before and after design
• Examples of a single group before and after design
• Risk of bias with a single group before and after design

Types of Evidence for Policy Evaluation – Matched Comparison Designs

• The principles of matching
• Propensity Score Matching
• Examples of Propensity Score Matching for policy making
• Risk of bias with Propensity Score Matching

Types of Evidence for Policy Evaluation – Difference-in-Difference Analysis

• Single difference and double difference
• Risk of bias with difference-in-difference analysis
• Examples of difference-in-difference analysis in policy making

Types of Evidence for Policy Evaluation – Regression Discontinuity Design

• The logic of regression discontinuity design
• Examples of regression discontinuity design in policy making
• Risk of bias with regression discontinuity design

Types of Evidence for Policy Evaluation - Randomised Controlled Trials

• How RCTs establish a strong counterfactual
• Risk of bias with RCTs
• Examples of RCTs in policy making

Types of Evidence for Policy Evaluation – Implementation Evaluation

• Delivery Trajectories
• Developing a Delivery Plan
• Undertaking a Priority Review

Types of Evidence for Policy Evaluation – Economic Appraisal

• The Green Book
• Cost-Benefit, Cost-Effectiveness and Cost-Utility Analysis
• Monetising cost and benefits
• Establishing New Present Value and Equivalent Annual Net Cost to Business

Presenting Evaluation Evidence

• The 1:3:25 format
• Writing clearly in non-technical language
• The Title
• Key messages for policy makers (one page)
• Executive Summary (three pages)
• Policy background and context
• Problem description
• Policy options
• Implementation and delivery issues
• Resource implications
• Conclusions and recommendations