Leading Evaluation - the story so far...

Thu 1st Feb – by Liz McIntyre

The case for good leadership in the NHS is generally well understood and the benefits are clear in terms of safer services, better patient outcomes, more engaged staff, as well promoting a more inclusive approach and better overall performance. However its harder to evidence that leadership development – in all its guises – creates better leaders who go on to deliver the better services but it’s important that we continually strive to develop critical insights into this causal relationship.

Each year, our Network of NHS Leadership Academies (NoA) is responsible for supporting the development of thousands of NHS leaders, through national and locally bespoke interventions but we face the challenge of demonstrating the effect of this substantial investment on the front line services that the NHS provides. Even more so, in the current NHS climate of over-stretched and financially constrained services, where every penny needs to be spent wisely, an evidence base which offers proof of high impact development interventions is highly prized.

Nonetheless, the evaluation of leadership development is renowned for being complex and difficult, with many variables affecting the ability to establish causation and identify patterns, let alone create generalizable learning which robustly outlines the context within which each intervention will work. Consequently evaluation is often used retrospectively as a means to offer an assurance of the quality of the intervention rather than used prospectively as a vehicle to improve the impact and effectiveness of the intervention. 

So, in late 2015 the NoA committed to an evaluation strategy which would: 

“Develop a culture of comprehensive and consistent evaluation which is embedded into all NHS leadership development activity and where leadership development teams are enabled to become highly competent in the process of strategic evaluation”.

From this strong desire to collaborate across the whole NoA, and through the curation of high quality academic research coupled with a strong service improvement and leadership development practitioner focus, the “Leading Evaluation, Development and Research (LeaDER) framework” was created.

Whilst originally focused on Leadership development, the LeaDER framework offers a single unifying framework for evaluating any improvement intervention where there is an importance for alignment with impact outcomes, strong understanding of context and a desire to unpick and test the assumptions that practitioners make about how and why an intervention will be effective. 

LeaDER utilises the concept of “Theory of Change” to predict how the intervention will work by articulating a common vision of what the impact will be at an individual, organisational and system level as appropriate to the planned intervention. It suggests that evaluation is reframed as an integral, prospective element of the intervention and planned from the beginning as a methodological practice.

leader framework

 

 

 

 

 

 

 

 

 

 

 

 

During 2016, 100 separate evaluations from across the country were undertaken. This information was then analysed as an aggregated sample to create the first ever study of this size into Leadership development. As well as suggesting ways in which the framework could be improved, the preliminary results of this study have given some helpful insights into the nature of leadership development in the NHS and even more importantly how we can finesse our practice to maximise the impact from an investment in leadership development. 

LeaDER is currently helping us to understand the nature of highly effective leadership development by enabling us to see how:

  • Prospective evaluation is changing the focus of our interventions. There have been a number of examples where, when the assumptions underpinning the drivers for change were exposed, the recommended intervention or the method of measuring the anticipated change was modified with significantly better outcomes.  On this basis, strategic evaluation using LeaDER is increasingly helping to strengthen the links between leadership development, service improvement and organisational development.
     
  • Better aligned programmes maximise stakeholder value.  Sometimes practitioners jump to solutions to an issue without fully appreciating the complex and dynamic nature of the problem. LeaDER is encouraging practitioners to take a step back and consider how solutions really fix the difficulties that instigated the need for an intervention. We are also challenging our own thinking about how we measure those changes, what evidence we can gather, and how we critique and analyse our outputs. When practitioners understand that alignment between these factors is critical to success then interventions become more effective at offering a return on stakeholder expectations.

    Issues of learning transfer can be managed better. Kirkpatrick and Kirkpatrick (2006) described a common challenge in education in how to develop programmes which hope to have an impact on organisational culture or elicit an individual behavioural change and yet outcomes fail to go beyond primary learning objectives. 

    Understanding how and why learning isn’t transferred into practice or sustained for long enough to realise its benefits is a key insight but it’s not enough to simply understand the issue. LeaDER is helping us to theorise about potential solutions and facilitate the testing of new ways to develop training which incorporates and brings closer together, both ends of the theory practice gap. 
     
  • Stronger feedback loops can be created. By establishing a single, consistent and nationwide evaluation framework we are creating a feedback cycle. This encourages us to learn and improve through experience and insight over time rather than reinforce a ‘pass or fail’ approach, thus enabling learning about how we can do our work better in a more supportive, collaborative environment. In time, the LeaDER framework offers us a way of promoting innovation in education and real workplace transformation.

    We need to work in partnership. All development has a single over-riding purpose and that is to improve practice. The focus of development could be on improving an individual’s skills, knowledge or behaviour or alternatively there might be an expectation of an improvement in organisational performance, processes, financial standing or in the way that the system works collaboratively along a patient pathway. Therefore development is inextricably linked to practice and the providers of development will need to create partnerships with clinicians, managers and others working within health and care if they are to ensure that this purpose remains core to the training activity planned. LeaDER is helping us to see how the closeness and strength of these partnerships can also help to increase the sustainable effectiveness and impact of our interventions.
  • Critics such as Kellerman (2012) observe that “leadership programmes tend to proliferate without objective assessment” and such programmes should subject themselves to critical analysis of their impact. Through developing the skills of our teams to evaluate objectively, LeaDER presents the NHS with an exciting opportunity to do this and to kick start a radical overhaul of the way in which leadership development is designed, implemented and sustained in the crucible of practice. Launch of version 2 of the framework is planned for April, 2018.

    Author: Lyndsay Bunting 

    Deputy Director and NoA Strategic Evaluation and Impact Assessment Lead

    For further information please contact Lyndsay.bunting@nottshc.nhs.uk 

References

•The Network of NHS Leadership Academies (2016)“Leading Leadership Development, Evaluation and Research- The LeaDER Framework” Version 1.2.

•Kirkpatrick D and Kirkpatrick J (2006) “Evaluating training programmes”. 3rd Edition. Berrett-Koehler Publishers Inc.

•Kellerman (2012)- “The END of leadership” HarperCollins.

FOOT NOTE

The NHS Network of Academies is an alliance of 10 Local Leadership Academies (LLA’s) working alongside the National Leadership Academy to create a dynamic community of practice. This community of practice enables the leadership development of thousands of individual leaders, teams and organisations every year through a diverse offer of interventions from coaching to bespoke in house development