How to … assess the impact of evidence-informed interventions

What difference does the RLC approach make? Chris Brown provides a model for assessing impact.

Workshop four of the RLC approach focuses on impact. With this ‘HOW TO’ I show how the Dialogic Model of Impact (DMI) (Brown, 2017) can be used to measure the impact of evidence-informed interventions. DMI was designed with the specific purpose of evaluating the impact of Renfrew County Catholic District School Board’s roll-out and expansion of the Through their Eyes: Documenting Literacy and Learning in Kindergarten approach to teaching and learning. DMI has general applicability however, and can be used to measure the impact of any innovation.

Create an account to read this article

£7.00+ VAT

One-off purchase

Purchase and Download today

Register for free

No Credit Card required

Register for free
Access to 3 free articles
Free TeachingTimes Report every month

array(8) {
  ["distinct_key"]=>
  string(10) "post#59757"
  ["title"]=>
  string(64) "How to … assess the impact of evidenced-informed interventions"
  ["categories"]=>
  array(2) {
    [0]=>
    string(21) "'How to do it' Guides"
    [1]=>
    string(24) "Professional Development"
  }
  ["url"]=>
  string(91) "https://www.teachingtimes.com/how-to-assess-the-impact-of-evidenced-informed-interventions/"
  ["post_date_timestamp"]=>
  int(1558370816)
  ["objectID"]=>
  string(12) "post-59757-0"
  ["_snippetResult"]=>
  array(2) {
    ["excerpt"]=>
    array(2) {
      ["value"]=>
      string(127) "What difference does the Research Learning Community approach make? Chris Brown provides a model for assessing impact."
      ["matchLevel"]=>
      string(7) "partial"
    }
    ["content"]=>
    array(2) {
      ["value"]=>
      string(363) "With this ‘HOW TO’  I show how the Dialogic Model of Impact (DMI) (Brown, 2017) can be used to measure the impact of evidence-informed interventions. DMI was designed with the specific purpose of evaluating the impact of Renfrew County Catholic District School Board’s roll-out and expansion of ..."
      ["matchLevel"]=>
      string(7) "partial"
    }
  }
  ["_highlightResult"]=>
  array(5) {
    ["title"]=>
    array(4) {
      ["value"]=>
      string(100) "How to … assess the impact of evidenced-informed interventions"
      ["matchLevel"]=>
      string(7) "partial"
      ["fullyHighlighted"]=>
      bool(false)
      ["matchedWords"]=>
      array(4) {
        [0]=>
        string(6) "assess"
        [1]=>
        string(6) "impact"
        [2]=>
        string(8) "informed"
        [3]=>
        string(13) "interventions"
      }
    }
    ["author"]=>
    array(1) {
      [0]=>
      array(3) {
        ["value"]=>
        string(11) "Chris Brown"
        ["matchLevel"]=>
        string(4) "none"
        ["matchedWords"]=>
        array(0) {
        }
      }
    }
    ["excerpt"]=>
    array(4) {
      ["value"]=>
      string(127) "What difference does the Research Learning Community approach make? Chris Brown provides a model for assessing impact."
      ["matchLevel"]=>
      string(7) "partial"
      ["fullyHighlighted"]=>
      bool(false)
      ["matchedWords"]=>
      array(1) {
        [0]=>
        string(6) "impact"
      }
    }
    ["content"]=>
    array(4) {
      ["value"]=>
      string(5825) "With this ‘HOW TO’  I show how the Dialogic Model of Impact (DMI) (Brown, 2017) can be used to measure the impact of evidence-informed interventions. DMI was designed with the specific purpose of evaluating the impact of Renfrew County Catholic District School Board’s roll-out and expansion of the Through their Eyes: Documenting Literacy and Learning in Kindergarten approach to teaching and learning. DMI has general applicability however, and can be used to measure the impact of any innovation. Key componentsUnderpinning DMI is the idea that impact comprises three key components or characteristics: the content of an innovation; the processes involved as part of the innovation; and the context in which the innovation is situated. The first of these, the ‘content’ characteristics, considers the ‘what’ of the innovation: for example, the new knowledge, skills or understanding that it is hoped the innovation will deliver. Second, ‘processes’ concern the ‘how’ – how will the innovation be planned, organized, delivered, continuously supported and so on. Finally ‘context’ characteristics are the ‘who’, ‘when’, ‘where’ and ‘why’ of the innovation – e.g. those innovating, the recipients of the innovation, as well as the systems, cultures and wider environment into which the innovation will be introduced. Simultaneously, evaluations should also consider the ‘quality’ of the innovation, the extent to which it has changed teacher knowledge and practice and, as a result, how the innovation has affected student outcomes (Guskey, 2000).Measuring commitment to the RLC approachAlongside these components it is also useful to consider how the recipients for a new innovation have engaged with it: in other words to examine scales or rubrics which examine the extent to which people feel invested in or concerned in relation to any new approach; the extent to which they will employ the new approach or even the ways in which they do so; and, as a result, the likelihood that any new approach will impact on practice. These highlight that innovators need to explicitly consider the following features:what specific problem the innovation is designed to address and the wider context encapsulating this problem;how recipients might come to learn about the innovation; the ways in which recipients might use the innovation and whether different interactions, engagement in different activities, or even different understandings of the problem or context, might impact on learning and understanding, as well as the levels of practice and pupil outcome change that are ultimately achieved. By spotlighting the why and how of an innovation through the use of the engagement indicators, the logic of its operation is necessarily brought to the fore as well as an understanding of what might help or hinder its roll-out. Impact domainsBringing together content, processes, context and quality, along with the how and why of an innovation’s operation results in the identification of the DMI’s eight domains of impact: The context in which the school or setting is situatedThe problem or driver for innovationDetail on the innovation and how it was intended to result in changeActivities and interactions related to the introduction and roll-out of the approachLearning that results from engaging in these activities/results from interactionsChanges in behaviour (and the extent to which something is being used): What difference have behavioural changes made?Reframing value: reassessing what is possible in relation to the innovationIntention and outcomesThe DMI approach is grounded in the very real possibility that there may be differences between how innovations were intended to be implemented and used, and how they actually are implemented and used. This highlights the iterative nature of impact: that any measure of impact is only a snapshot in time. This means that when seeking to understand the extent of positive change an impact has had, we also have the opportunity to examine how this change can be enhanced further. That impact can and does change over time implies that measuring it using purely linear approaches (e.g. simply by measuring impact as a snapshot of change across impact levels) will be less effective than approaches that examine impact dynamically, and from a multitude of perspectives.So, it is likely to be more fruitful to use impact domains to:first explore what innovators hoped to achieve and how they hoped to achieve it; then to use impact domains to examine recipients’ perspectives on the problem to be addressed, the innovation used to address it, their experience of its roll-out and how moving forward the implementation of the innovation could be improved; andfinally, based on the impact data collected, to use the impact domains once again to help innovators determine key areas for improvement as well as what might be realistically expected from the innovation moving forward. This type dynamic approach can be achieved by employing multiple research conversations, by comparing the findings that emerge from these conversations and then by employing this data as part of the process of improvement. The focus of these research conversations should therefore be to explore:What was supposed to happen (determined by using the impact domains to elicit information from innovators); What happened (ascertained by using the impact domains to elicit information from the recipients of the innovation);How can the approach be"
      ["matchLevel"]=>
      string(7) "partial"
      ["fullyHighlighted"]=>
      bool(false)
      ["matchedWords"]=>
      array(4) {
        [0]=>
        string(6) "impact"
        [1]=>
        string(8) "evidence"
        [2]=>
        string(8) "informed"
        [3]=>
        string(13) "interventions"
      }
    }
    ["categories"]=>
    array(2) {
      [0]=>
      array(3) {
        ["value"]=>
        string(21) "'How to do it' Guides"
        ["matchLevel"]=>
        string(4) "none"
        ["matchedWords"]=>
        array(0) {
        }
      }
      [1]=>
      array(3) {
        ["value"]=>
        string(24) "Professional Development"
        ["matchLevel"]=>
        string(4) "none"
        ["matchedWords"]=>
        array(0) {
        }
      }
    }
  }
}

'How to do it' Guides

How to … assess the impact of evidenced-informed interventions

20 May 2019

array(8) {
  ["distinct_key"]=>
  string(10) "post#59766"
  ["title"]=>
  string(58) "How to … select the right participants to engage in RCLs"
  ["categories"]=>
  array(2) {
    [0]=>
    string(21) "'How to do it' Guides"
    [1]=>
    string(24) "Professional Development"
  }
  ["url"]=>
  string(85) "https://www.teachingtimes.com/how-to-select-the-right-participants-to-engage-in-rcls/"
  ["post_date_timestamp"]=>
  int(1558370909)
  ["objectID"]=>
  string(12) "post-59766-1"
  ["_snippetResult"]=>
  array(2) {
    ["excerpt"]=>
    array(2) {
      ["value"]=>
      string(111) "Chris Brown gives guidance on how to identify change leaders and opinion formers to facilitate the RLC process."
      ["matchLevel"]=>
      string(4) "none"
    }
    ["content"]=>
    array(2) {
      ["value"]=>
      string(424) "... opinion.  Once identified these senior leaders and opinion formers then go through the cycle of enquiry activity detailed above. Here they engage with research, develop evidence informed interventions, roll out evidence-informed interventions within their school and then assess the impact of their efforts. These key tasks are outlined in ..."
      ["matchLevel"]=>
      string(4) "full"
    }
  }
  ["_highlightResult"]=>
  array(5) {
    ["title"]=>
    array(3) {
      ["value"]=>
      string(58) "How to … select the right participants to engage in RCLs"
      ["matchLevel"]=>
      string(4) "none"
      ["matchedWords"]=>
      array(0) {
      }
    }
    ["author"]=>
    array(1) {
      [0]=>
      array(3) {
        ["value"]=>
        string(11) "Chris Brown"
        ["matchLevel"]=>
        string(4) "none"
        ["matchedWords"]=>
        array(0) {
        }
      }
    }
    ["excerpt"]=>
    array(3) {
      ["value"]=>
      string(111) "Chris Brown gives guidance on how to identify change leaders and opinion formers to facilitate the RLC process."
      ["matchLevel"]=>
      string(4) "none"
      ["matchedWords"]=>
      array(0) {
      }
    }
    ["content"]=>
    array(4) {
      ["value"]=>
      string(3245) "to identify names in response to the question ‘to whom do you turn as reliable sources of expertise in terms of teaching and learning’. Teachers are then asked to indicate how often they engaged with specific colleagues using on a 5-point scale ranging from 1 (1-2 times a week) to 5 (Not at all). In addition to the frequency of their teaching and learning related interactions, teachers are asked to assess the quality of such interactions by reflecting their degree of usefulness1. Relationships recorded as being ‘high quality’, i.e. more frequent and more useful (Carley and Krackhardt, 1999) are then examined.Social networks represent the connections that exist between a group of individuals, for instance those situated within a school. Social Network Analysts are often concerned with how networks are used by groups to share and access knowledge, expertise, advice or other forms of social capital. To assess this they use social network surveys to identify the existence and nature of networks and how they function. Typically such surveys will ask respondents to name the colleagues they engage with, to state why they engage with particular individuals and to examine the quality of those interactions.  A similar process is repeated in order to ascertain the value staff place on the opinions proffered by their colleagues (‘to whom do you turn when seeking out opinions on new teaching practices’; here responses are measured on a ‘yes’/’no’ scale).  These results are combined to identify potential galvanizers – influencers supportive of new initaitives and likely to be able to promote the adoption of new practices. They will be those teachers with high scores in terms of both the frequency and quality of their advice giving relationships with teachers, and who were turned to by more colleagues for their opinion.  Once identified these senior leaders and opinion formers then go through the cycle of enquiry activity detailed above. Here they engage with research, develop evidence informed interventions, roll out evidence-informed interventions within their school and then assess the impact of their efforts. These key tasks are outlined in the next three ‘HOW TOs’.Professional Learning Task: Mapping knowledge flowWe use social network analysis as a way of identifying opinion formers. To replicate this approach the following exercise gets you to think about how knowledge flows around your school. To begin with take a sheet of A3 paper. Then allocate different shapes for different roles within your school. For example circles could represent SLT members, triangles could represent middle leaders and stars could represent teachers. Colour can be used to delineate teams, e.g. year groups or departments. Thinking of the question, To whom do teachers turn as reliable sources of expertise in terms of teaching and learning, use lines and arrows to show how knowledge flows within your school. What does this tell you about who should be involved in RLC activity?     1. The quality of ties were measured using a 5-point scale, ranging from 1 (Not at all useful) to 5 (Very useful)."
      ["matchLevel"]=>
      string(4) "full"
      ["fullyHighlighted"]=>
      bool(false)
      ["matchedWords"]=>
      array(5) {
        [0]=>
        string(6) "assess"
        [1]=>
        string(6) "impact"
        [2]=>
        string(8) "evidence"
        [3]=>
        string(8) "informed"
        [4]=>
        string(13) "interventions"
      }
    }
    ["categories"]=>
    array(2) {
      [0]=>
      array(3) {
        ["value"]=>
        string(21) "'How to do it' Guides"
        ["matchLevel"]=>
        string(4) "none"
        ["matchedWords"]=>
        array(0) {
        }
      }
      [1]=>
      array(3) {
        ["value"]=>
        string(24) "Professional Development"
        ["matchLevel"]=>
        string(4) "none"
        ["matchedWords"]=>
        array(0) {
        }
      }
    }
  }
}

'How to do it' Guides

How to … select the right participants to engage in RCLs

20 May 2019

array(8) {
  ["distinct_key"]=>
  string(10) "post#43664"
  ["title"]=>
  string(83) "How to … select the right participants to engage in Research Learning Communities"
  ["categories"]=>
  array(2) {
    [0]=>
    string(21) "'How to do it' Guides"
    [1]=>
    string(24) "Professional Development"
  }
  ["url"]=>
  string(61) "https://www.teachingtimes.com/pdt192-selectingtherightpeople/"
  ["post_date_timestamp"]=>
  int(1497960766)
  ["objectID"]=>
  string(12) "post-43664-1"
  ["_snippetResult"]=>
  array(2) {
    ["excerpt"]=>
    array(2) {
      ["value"]=>
      string(111) "Chris Brown gives guidance on how to identify change leaders and opinion formers to facilitate the RLC process."
      ["matchLevel"]=>
      string(4) "none"
    }
    ["content"]=>
    array(2) {
      ["value"]=>
      string(424) "... opinion.  Once identified these senior leaders and opinion formers then go through the cycle of enquiry activity detailed above. Here they engage with research, develop evidence informed interventions, roll out evidence-informed interventions within their school and then assess the impact of their efforts. These key tasks are outlined in ..."
      ["matchLevel"]=>
      string(4) "full"
    }
  }
  ["_highlightResult"]=>
  array(5) {
    ["title"]=>
    array(3) {
      ["value"]=>
      string(83) "How to … select the right participants to engage in Research Learning Communities"
      ["matchLevel"]=>
      string(4) "none"
      ["matchedWords"]=>
      array(0) {
      }
    }
    ["author"]=>
    array(1) {
      [0]=>
      array(3) {
        ["value"]=>
        string(11) "Chris Brown"
        ["matchLevel"]=>
        string(4) "none"
        ["matchedWords"]=>
        array(0) {
        }
      }
    }
    ["excerpt"]=>
    array(3) {
      ["value"]=>
      string(111) "Chris Brown gives guidance on how to identify change leaders and opinion formers to facilitate the RLC process."
      ["matchLevel"]=>
      string(4) "none"
      ["matchedWords"]=>
      array(0) {
      }
    }
    ["content"]=>
    array(4) {
      ["value"]=>
      string(4214) "support, As such they are able to act as conduits for both mobilizing opinion and the adoption of new practices.Setting up an RLC- Identifying the key roles RLCs typically comprise eight to 12 teachers from four to six schools, although this number could be higher. What is important however is that each school should nominate a senior leader (e.g. a school leader/headteacher, assistant head or the deputy head of schools) and an opinion former to participate. Identifying the former is relatively easy. However, in order to identify opinion formers, it is helpful to administer a social network survey to all teaching staff within schools, including school leaders, classroom teachers, instructional support staff, etc. Social network data should be collected in two key areas. First we ask school staff to assess the frequency of their expertise related interactions with other colleagues of their school. This is assessed by asking teachers to identify names in response to the question ‘to whom do you turn as reliable sources of expertise in terms of teaching and learning’. Teachers are then asked to indicate how often they engaged with specific colleagues using on a 5-point scale ranging from 1 (1-2 times a week) to 5 (Not at all). In addition to the frequency of their teaching and learning related interactions, teachers are asked to assess the quality of such interactions by reflecting their degree of usefulness1. Relationships recorded as being ‘high quality’, i.e. more frequent and more useful (Carley and Krackhardt, 1999) are then examined.Social networks represent the connections that exist between a group of individuals, for instance those situated within a school. Social Network Analysts are often concerned with how networks are used by groups to share and access knowledge, expertise, advice or other forms of social capital. To assess this they use social network surveys to identify the existence and nature of networks and how they function. Typically such surveys will ask respondents to name the colleagues they engage with, to state why they engage with particular individuals and to examine the quality of those interactions.  A similar process is repeated in order to ascertain the value staff place on the opinions proffered by their colleagues (‘to whom do you turn when seeking out opinions on new teaching practices’; here responses are measured on a ‘yes’/’no’ scale).  These results are combined to identify potential galvanizers – influencers supportive of new initaitives and likely to be able to promote the adoption of new practices. They will be those teachers with high scores in terms of both the frequency and quality of their advice giving relationships with teachers, and who were turned to by more colleagues for their opinion.  Once identified these senior leaders and opinion formers then go through the cycle of enquiry activity detailed above. Here they engage with research, develop evidence informed interventions, roll out evidence-informed interventions within their school and then assess the impact of their efforts. These key tasks are outlined in the next three ‘HOW TOs’.Professional Learning Task: Mapping knowledge flowWe use social network analysis as a way of identifying opinion formers. To replicate this approach the following exercise gets you to think about how knowledge flows around your school. To begin with take a sheet of A3 paper. Then allocate different shapes for different roles within your school. For example circles could represent SLT members, triangles could represent middle leaders and stars could represent teachers. Colour can be used to delineate teams, e.g. year groups or departments. Thinking of the question, To whom do teachers turn as reliable sources of expertise in terms of teaching and learning, use lines and arrows to show how knowledge flows within your school. What does this tell you about who should be involved in RLC activity?Endnotes 1. The quality of ties were measured using a 5-point scale, ranging from 1 (Not at all useful) to 5 (Very useful)."
      ["matchLevel"]=>
      string(4) "full"
      ["fullyHighlighted"]=>
      bool(false)
      ["matchedWords"]=>
      array(5) {
        [0]=>
        string(6) "assess"
        [1]=>
        string(6) "impact"
        [2]=>
        string(8) "evidence"
        [3]=>
        string(8) "informed"
        [4]=>
        string(13) "interventions"
      }
    }
    ["categories"]=>
    array(2) {
      [0]=>
      array(3) {
        ["value"]=>
        string(21) "'How to do it' Guides"
        ["matchLevel"]=>
        string(4) "none"
        ["matchedWords"]=>
        array(0) {
        }
      }
      [1]=>
      array(3) {
        ["value"]=>
        string(24) "Professional Development"
        ["matchLevel"]=>
        string(4) "none"
        ["matchedWords"]=>
        array(0) {
        }
      }
    }
  }
}

'How to do it' Guides

How to … select the right participants to engage in Research Learning Communities

20 Jun 2017

array(8) {
  ["distinct_key"]=>
  string(10) "post#43662"
  ["title"]=>
  string(50) "How to… facilitate Research Learning Communities"
  ["categories"]=>
  array(2) {
    [0]=>
    string(24) "Professional Development"
    [1]=>
    string(8) "Research"
  }
  ["url"]=>
  string(61) "https://www.teachingtimes.com/pdt192-howtofacilitateresearch/"
  ["post_date_timestamp"]=>
  int(1497960587)
  ["objectID"]=>
  string(12) "post-43662-0"
  ["_snippetResult"]=>
  array(2) {
    ["excerpt"]=>
    array(2) {
      ["value"]=>
      string(254) "Research Learning Communities represent a creative, dynamic approach to professional learning. They have been pioneered by Chris Brown from the Institute of Education, University of London (IoE, UCL) and comprise a number of distinctive ingredients. RCLs"
      ["matchLevel"]=>
      string(4) "none"
    }
    ["content"]=>
    array(2) {
      ["value"]=>
      string(405) "... crucial importance of involvingthe “right” people. Next is guidance on how to engage effectively with research, followed by how todevelop, trial and roll-out evidenced-informed interventions. The concluding contribution gives acomprehensive approach on how to assess the impact of these interventions.For each HOW TO there are Professional Learning ..."
      ["matchLevel"]=>
      string(7) "partial"
    }
  }
  ["_highlightResult"]=>
  array(5) {
    ["title"]=>
    array(3) {
      ["value"]=>
      string(50) "How to… facilitate Research Learning Communities"
      ["matchLevel"]=>
      string(4) "none"
      ["matchedWords"]=>
      array(0) {
      }
    }
    ["author"]=>
    array(1) {
      [0]=>
      array(3) {
        ["value"]=>
        string(11) "Chris Brown"
        ["matchLevel"]=>
        string(4) "none"
        ["matchedWords"]=>
        array(0) {
        }
      }
    }
    ["excerpt"]=>
    array(3) {
      ["value"]=>
      string(254) "Research Learning Communities represent a creative, dynamic approach to professional learning. They have been pioneered by Chris Brown from the Institute of Education, University of London (IoE, UCL) and comprise a number of distinctive ingredients. RCLs"
      ["matchLevel"]=>
      string(4) "none"
      ["matchedWords"]=>
      array(0) {
      }
    }
    ["content"]=>
    array(4) {
      ["value"]=>
      string(1461) "29➜ How to…How to…facilitate Research LearningCommunitiesResearch Learning Communities represent a creative, dynamic approach to professional learning.They have been pioneered by Chris Brown from the Institute of Education, University of London (IoE,UCL) and comprise a number of distinctive ingredients. RCLs entail people coming together froma network of schools. These participants involve both a senior leader and an evidence or researchchampion who maintain an explicit focus on learning, built upon existing academic research,together with research knowledge generated within the school. This professional learning is used,through specific interventions, to secure explicit improvements in teaching and learning.Chris Brown has created the following HOW TO guidance to Research Learning Communities.The first piece explores their potential and the second looks at the crucial importance of involvingthe “right” people. Next is guidance on how to engage effectively with research, followed by how todevelop, trial and roll-out evidenced-informed interventions. The concluding contribution gives acomprehensive approach on how to assess the impact of these interventions.For each HOW TO there are Professional Learning Tasks which enable readers to consider andapply the guidance to their own context and practice. In this way we hope you too get the ResearchLearning Communities bug!"
      ["matchLevel"]=>
      string(4) "full"
      ["fullyHighlighted"]=>
      bool(false)
      ["matchedWords"]=>
      array(5) {
        [0]=>
        string(6) "assess"
        [1]=>
        string(6) "impact"
        [2]=>
        string(8) "evidence"
        [3]=>
        string(8) "informed"
        [4]=>
        string(13) "interventions"
      }
    }
    ["categories"]=>
    array(2) {
      [0]=>
      array(3) {
        ["value"]=>
        string(24) "Professional Development"
        ["matchLevel"]=>
        string(4) "none"
        ["matchedWords"]=>
        array(0) {
        }
      }
      [1]=>
      array(3) {
        ["value"]=>
        string(8) "Research"
        ["matchLevel"]=>
        string(4) "none"
        ["matchedWords"]=>
        array(0) {
        }
      }
    }
  }
}

Professional Development

How to… facilitate Research Learning Communities

20 Jun 2017