1: Introduction to evaluation planning.

​

The following resources give a general overview of key considerations and steps to follow in planning an evaluation. They provide a good place to start before going on to detailed evaluation planning.

 

Vinal K Karania. (2017) Service Evaluation: An Introductory Guide. Age UK.

An Age UK resource, this guide provides a straightforward introduction to planning an evaluation. It defines two types of evaluation (process and impact evaluations) and provides 7 questions to consider in planning and designing an evaluation. It has a useful glossary and link to further information.

​

 

Manage an Evaluation or Evaluation Planning. Better Evaluation.

This publication covers engaging stakeholders, overseeing evaluations, decision-making, determining what resources are needed, document management planning and development of an evaluation plan. There is a small section on how to develop evaluation capacity.

​

 

Guidelines for Good Practice in Evaluation. UK Evaluation Society.

This document provides brief guidance written by the UK Evaluation Society on different aspects of evaluation. These guidelines have been written ‘…for use by the novice and the experienced alike’ and set out principles to inform evaluation practice, practical guidance for evaluation practice and ethical considerations.

​​

Data Cymru’s Introductory Guide to Evaluation.

This is an accessible, comprehensive and succinct web guide to planning and carrying out evaluations. It clearly sets out the evaluation process, describes types of evaluation, evaluation frameworks and models (i.e. logic models and theory of change) and has a useful glossary. There is a direct link to the HM Treasury Magenta Book, which is a key resource for planning evaluations.

Data Cymru’s guide is available in Welsh.

​

 

2: Developing a question is the starting point for a research/ evaluation plan.

 

Holm-Hansen, C. (2006) Choosing the Right Questions: Tips for Conducting Program evaluation.  Wilder Research.

This four-page resource provides information on how to determine the key evaluation questions to explore. It covers outcome, process and satisfaction evaluations and provides practical guidance.

​

 

Wingate, L., & Schroeter, D. (2007). Evaluation questions checklist for program evaluation. Retrieved from http://wmich.edu/evaluation/checklists 

​

This resource provides an accessible and practical aid to developing relevant evaluation questions. It covers how to determine evaluation questions that are ‘pertinent’, ‘reasonable’, ‘specific’ and ‘answerable’. It also includes a list of other useful resources.

​

​

Good Evaluation Questions: A Checklist to Help Focus Your Evaluation.  Centre for Disease Control and Prevention, USA.

​

“The most serious mistakes are not being made as a result of wrong answers. The truly dangerous thing is asking the wrong question.” 

-Peter Drucker

​

Starting with a quote from Peter Drucker, this two-page resource has been developed from practice wisdom and knowledge of the evaluation literature. It includes a concise checklist to assist the development of a suitable evaluation question. This covers stakeholder engagement and the fit, relevance and feasibility of the evaluation.

​

​

Co-Production Network for Wales (2019). Measuring what matters.

​

Measuring what matters evaluation that asks the right questions of the right people in the right way. 

​

This publication produced by the Co-Production Network for Wales emerged from a series of seminars in Wales called ‘Tents around the Campfire’ which explored different perspectives on evaluation. This publication is a comprehensive tool to assist you to get to the heart of your evaluation needs and identify the right questions and data collection methods to use. It is based on 4 steps.

​

STEP 1: Select your evaluation question

STEP 2: Consider your people, purpose and process

STEP 3: Identify your data collection options (selecting from 52                  named methods)

STEP 4: Check and record your decision

         The tool will assist you to move from what you want to do, to           the data collection methods that will enable this to happen.

​

 

Preskill, H. and Jones, N. (2009) Practical Guide for Engaging Stakeholders in Developing Evaluation Questions. From the Robert Wood Johnson Foundation Evaluation Series.

​

This resource has an accessible five-step framework for involving stakeholders in setting evaluation questions. It covers different engagement methods to use in planning as well as strategies for managing challenges. It includes template sheets to apply the resources used in the framework.

​

 

3: Create a logic model to decide what evidence you need to collect to answer your question.

 

The Strategy Unit. (2016) Using Logic Models in Evaluation. NHS Midlands and Lancashire Commissioning Unit.

This briefing paper gives a brief introduction to logic models and the benefits of them as well as guidance for using logic models.

​

 

Developing a Logic Model. Evaluation Support Scotland.

This resource provides an accessible guide to logic models. It takes the reader through 2 types of logic models ‘The Weavers Triangle’ and the ‘Wisconsin Model’, giving examples of each type and how to produce them.

​

Hills. D. (2010) Logic Mapping: Hints and Tips. For Better Transport Evaluation. The Tavistock Institute.

​

Although this was developed for the Department of Transport, the content is applicable to other contexts. It provides an overview of how logic models are used in evaluations. Chapter 2 provides a step by step guide to logic mapping.

​

​

 

4.Evaluation Planning. Deciding what needs to be in your plan.

 

Evaluation Support Scotland have a detailed website with evaluation guides. They have produced a series of guides that can be used to plan an evaluation.

1a. Setting Outcomes.

1b. Working out what to measure (setting indicators for your outcomes.

2a  Designing evidence collection methods.

​

​

Better Evaluation. Rainbow Framework

​

Better Evaluation an international collaboration working to ‘improve evaluation theory and practice by sharing information about evaluation options (methods, strategies, processes) and approaches (collections of methods)’. The Rainbow Framework organises 34 evaluation options (methods, strategies and processes) by the 7 colours of the rainbow. Its aim is to assist in decision making about all aspects of evaluation planning.

​

 

Evaluation Practice Handbook.  World Health Organisation.

​

Although this has been written for a different audience to Welsh local authorities, Chapter 2 provides a useful step by step guide to planning an evaluation.

​

​

Evaluation Toolkit. Life Changes Trust.

​

This toolkit is a web resource provided by the Life Changes Trust. It provides comprehensive coverage of all aspects of evaluation including an overview of what is evidence, planning activities and outcomes, evaluation design, data collection methods, logic models and types of data and analysis. There is a section on matching evaluation questions to data collection methods. The toolkit can also be downloaded as a pdf. There is a valuable search engine on the home page and a glossary section.

​

 

Evaluation Plan Workbook. Innovation Network: Transforming Evaluation for Social Change.

​

This has detailed guidance for writing an evaluation plan, including clear guidance on indicators. It includes templates for an evaluation plan.

​

 

5. Evidence Collection.

​

You may find it helpful to refer to guidance about matching data collection methods to evaluation questions.

 

Age UK have developed a decision tree to help select the appropriate method for collecting evidence.

​

 

Life Changes Trust in the Evaluation Toolkit has a section on matching data collection methods to evaluation questions.

​

 

Once you have decided what data collection methods you are going to use you might want to find more guidance about how to use these methods.

 

If you are going to collect numerical data, you can access further resources using this link: Collecting and Analysing Numerical Data.

​

If you are collecting qualitative (non-numerical) data, you can access further resources using this link:

Collecting and Analysing Qualitative (Non-Numerical) Data.

 

 

 

 

 

 

 

6. Good practice in conducting evaluations.

These questions will test whether the evaluation plan is in line with good practice.

​

Have you involved all the relevant people in planning?

 

Who Wants to Know? Tips for Involving Stakeholders in Your Program Evaluation.

Cheryl Hosley for Wilder Research.

This contains four pages of practical guidance about involving stakeholders (i.e. relevant people) in planning the research.

​

 

Have you considered if your plan meets the relevant ethical requirements?

​

Social Care Wales Code of Professional Practice for Social Care.

The standards in the Code of Practice need to be applied when planning an evaluation to ensure that all ethical requirements are met.

​

 

Inspiring Impact.

This is an accessible guide to some of the research ethics and data protection requirements which apply when carrying out an evaluation.

​

 

Have you, as far as possible, planned a reliable, valid evaluation or research study that will have useful information for people outside the service?

Essentials of survey research and analysis. Ronald J Polland.

This has a short chapter on reliability and validity. It explains what things to consider when planning a valid and reliable study.

​

​

Have you ensured your evidence collection approach is suitable for all relevant members of the local society- i.e. is your approach inclusive?

​

Measuring what matters. Evaluation that asks the right questions of the right people in the right way (also referenced as: Measuring what matters (a thinking tool) Co-Production Network for Wales.

This guide provides information about how you can select methods that enable the people you are most interested in to participate, including how to adapt methods to make them more inclusive.

​

An alternative approach

 

Engaging in a peer review can offer challenge, shared learning and support from people who are facing the same problems. This forms another route to conducting an evaluation of how existing services are working.

​

There are limited resources available on how to conduct/ engage in a Peer Review but the guide below, written for the needs of services in London, contains helpful information on planning for an implementing a review. 

London Peer Review. A Guide for Boroughs Being Reviewed and Reviewers. Association of Directors of Adult Social Services in London.

​

​