• Evaluation of Learning & Teaching Innovation Projects

    This planning instrument is designed to stimulate reflection and action, and to enable you to work towards consistency in good practice with regard to evaluation of your learning and teaching innovation(s). The results from an evaluation can provide a foundation for your decision-making, and can be used to inform your practices and priorities.

  • Evaluation Planning

    An evaluation plan sets out the proposed details of an evaluation. It should include information about:

    • what the evaluation is trying to do (what is to be evaluated, the purposes of the evaluation and key evaluation questions)
    • how it will be done (data collection, data analysis, reporting results).

    (Betterevaluation.org)

    This instrument consists of six simple steps:     

    1. The Scope and Purpose of your evaluation
    2. The Stakeholders and Study audience for the evaluation
    3. Key Evaluation Questions
    4. Evaluation data collection and analysis
    5. Evidence of success
    6. Dissemination and reporting

    How to complete the evaluation planning exercise

    For each of the steps there are one or two questions which ask for your responses. Select the most applicable option(s) to your project or add your own response (‘other’).

    On completion of the exercise, a summary report will be generated for your reference and sent to the email address you provide. You can use this in one or more ways: to guide your evaluation; as a reminder during your project; as part of an application for a grant; to help complete formative or summative reports.

     

  • First, please complete some information about yourself:

    • Step 1. Purpose and Scope 

    • When thinking about the purpose of your evaluation, be careful not to confuse this with the aims of the project. Here are a number of questions to guide your reflection.

      • What exactly are you evaluating?
      • Why is the evaluation being done?
      • Are you basing the evaluation on any particular method, framework or approach?

      Next consider the results from the evaluation.

      • How will the information be used? This may be dependent on the audience for the evaluation so you may need to return to this question after completing Step 2. Considering usage may also help narrow down scope of the evaluation.
      • Who will evaluate this project? Are they suitably skilled?
      • What value will the evaluation process add to the project?

      Often evaluation studies are expected to be all things to all people, whereas the reality is that the Project Lead often has limited resources (time, funds, expertise) and thus can only focus on a limited range of purposes. This is particularly so in small, internally funded learning and teaching projects where evaluation is often overlooked or left too late to be of use due to insufficient planning (Huber & Harvey, 2013).

    • Step 1: Purpose and Scope

    • 1a. The purpose and scope of the evaluation is:


    • 1b. The information from the evaluation will be used:


    • Step 2. Stakeholders 

    • The Stakeholders of a project may not necessarily be the same as those for the evaluation. Some people prefer to consider this information before defining the purpose and scope as different stakeholders may have different requirements.

      Similarly, the stakeholders may not be the same group as the audiences for the evaluation information.

      To clarify:

      Stakeholders - individuals/groups/organisations that have something significant to gain or lose in relation to the project and therefore the evaluation.

      Audiences - are individuals/groups/organisations whose information needs are specifically being addressed in the evaluation. They will overlap considerably with the stakeholders, but should be viewed here as individuals and groups who receive information from the study and therefore should guide the manner in which information is produced and disseminated.

      For example, the funding body may be the audience for an evaluation report but it is unlikely they will be the primary stakeholder.

      Consider also how competing interests should be prioritised. In learning and teaching projects, students and staff are usually the key stakeholders although families, employers, and members of the wider community may also be legitimate stakeholders. It’s unlikely that an evaluation of a small internally funded project can meet the needs of all stakeholder groups so concentrate on one or two and align the purpose of the evaluation to their needs. This will ensure that the evaluation results have a higher probability of use (Patton, 1997).

    • Step 2: Stakeholders

    • 2a. The stakeholders of the project and of the evaluation are:

    •  

    • 2b The audience(s) for the evaluation results are:


    • Step 3. Questions 

    • What are the high level questions the evaluation will seek to answer? Think of this similarly to how you would pose a research question to frame a research study. The questions will likely be linked to the purpose of the evaluation and the stakeholder needs. Consider how these questions will be developed. Input from the stakeholders a critical friend or a reference group if applicable may be required.

      A further consideration is the context of the project. This information is critically important in evaluation, as it will facilitate a deeper understanding and explanation of the particular outcomes that are achieved and of the factors that have enabled them to occur. This information will also inform predictions about impacts of the project and influence the utilisation of the results (Chesterton & Cummings 2011; Alkin & Taut, 2003).

      The questions below are provided as summative and formative options. Depending on the size of the project, restrict your choice to a reasonable number. A two-year project may have about four or five key questions.

    • Step 3: Questions

    • 3a. The (formative) Key Evaluation Questions are:


    • 3b. The (summative) Key Evaluation Questions are:


    • Step 4. Data 

    • The data in an evaluation is collected primarily to address the key evaluation questions. Before making choices below, consider how the information will be collected and what data are most appropriate. For example staff and students will most likely be sources but will you interview all or sample them?

      What are the most appropriate methods of data collection? A matrix is useful in helping to map out the Key Evaluation Questions against potential data sources and helps identify possible overlaps in data collection and the development of more efficient processes.

      There are two main purposes for data analysis, description and explanation. The former provides information to the audience on the aims and outcomes of the project and how well they have been met. The latter provides evidence on the sustainability, reproducibility and transferability of the project.

      The data analysis allows for conclusions to be made about the success of a project. Some evaluators go on to then provide recommendations based on the findings.

      At this stage, thought should also be given into obtaining human ethics clearance, which will involve consideration of how any ethical issues will be addressed.

    • Step 4: Data

    • 4a. The data and evidence that will be collected are:


    • 4b. The data will come from: 


    • 4c. How the data will be analysed:


    • Step 5. Success 

    • Conducting an evaluation requires a value to be applied or judgment to be made. In order to do this, criteria need to be identified to judge whether certain standards have been met and to what extent. Such judgments will play a key role in any decision making that the evaluation is intended to inform. In general, judgments will be required for each key evaluation question (that you defined in step #3 above).

      Also consider what it would take for the project to be deemed successful. Furthermore, what items are considered as failure points?

      The examples below are written in the form of questions you can ask. Choose any or all that apply.

    • Step 5: Success

    • 5a. The criteria for judgment are:


    • 5b. The critical success factors are:


    • Step 6. Reporting 

    • Dissemination strategies are useful for informing the wider community of your findings but be careful not to conflate evaluation findings with project findings.

      In step 2, audiences for the evaluation study were identified. Each audience will have specific information needs and interests and reports should be tailored to these. For example the funding body may need information on how resources have been used, formative reporting could provide feedback to the project team on processes or design and engaging stakeholders may be required for greater utilisation of findings.

    • Step 6: Reporting

    • 6a. Dissemination strategies that will be used:


    • 6b. The amount of generalisability there may be:

    • 6c. Human ethics clearance:

    • 6d. Recommendations for the future:
  • Should be Empty: