Monitoring and evaluation
We have been evaluating our members' views of the operation and efficacy of the network as part of a long-term initiative to understand and demonstrate the impact of H&P. The overall aims of the monitoring and evaluation research are two-fold:
- Summative: the story of what has happened, stakeholders' experiences of H&P
- Formative: ideas for what could happen, how H&P could be shaped for the future to better work with stakeholders
The first phase of this research, which took place in 2010, involved detailed interviews with 28 network member historians.
The second phase, in 2011, focused on policy makers and journalists, two key H&P constituencies, whose views and experiences of the project we needed to understand better because of their important role in enabling us to fulfil H&P's ultimate goal of bringing high-quality history into the policy-making arena.
This phase involved interviews with 20 civil servants and journalists, conducted as were the first phase by external researchers. It also included a seminar bringing together some of the journalists and policy makers who participated in the interviews, and historians, to discuss and try to better understand the policy-making process and how impact on public policy can be measured.
Summary of findings: Phase 1
For many network members, H&P was important for 'just being there', providing and validating the link between historians, policymakers and the media. In this way they were highly supportive of the organisation, even if they considered themselves 'inactive members' or disagreed over the potential 'impact' of history and the role of historians in public life.
Network members had different experiences of H&P and individual expectations of what the organisation could deliver. Some were cynical as to whether politicians would ever listen to historians and others challenged H&P's definitions of 'policy'. Some network members were wary about the current use of the word 'impact,' espeically in terms of the forthcoming Research Excellence Framework (REF). (More information on the Research Excellence Framework and the potential impact it will have on historians can be found on the HEFCE website. The future of these reforms is currently uncertain following the change of government.) Nonetheless, despite these very different ideological perspectives, there was general agreement that H&P mattered.
This evaluation investigated network members' opinions on H&P services and communications, questioning what was more or less useful. It revealed that network members generally had a high level of awareness of H&P services and regularly accessed the website, and - in particular - read the newsletter. Network members were asked about their barriers to engagement with the organisation and in doing so uncovered issues relating to both time and geography.
Many historians talked about their busy schedules and their concern about the impact of the REF upon their timetables. Several apologised for not 'finding time' to become more involved, suggesting that time could be found if engagement with H&P became a 'good use of time'. Despite the website being 'geography-less', many felt that H&P was London-centric, particularly because many advertised events were based in London. Others talked about how H&P's definition of 'policy' was London-centric and focussed on national government which, as a result, excluded their own regionally-based policy engagement.
What was revealed during the evaluation was the high level of historians' engagement with the media, both with and without the support of H&P, although network members had better experiences with the media if it occurred with the support of H&P. Conversely, fewer network members had examples of contact with policymakers. Again, this contact was generally better structured and 'went well' if it occurred with the support of H&P. No network member kept H&P up to date with their 'other' media and policy work; either they felt H&P would not be interested, or they were unsure whether H&P would think it relevant.
Network members appraised the strengths and weaknesses of H&P and, despire their different expectations and experiences of the organisation - not to mention their debate over 'impact' - there was remarkable agreement both over the value of H&P's independence and for the need for more evidence of 'impact'. That most network members would pay a membership fee to continue H&P was remarkable; although many felt that do so would be at the expense of the independence which was so valued.
Summary of findings: Phase 2
H&P has been successful in linking journalists and policy makers to historians
12 of 20 respondents have been connected with an H&P historian and the majority of these contacts have gone 'very well' or 'well' and led to sustained relationships and planned follow-up activity. This is demonstrated by the following diagram:
Journalists and policy makers engaged with H&P have very positive views of history
The majority of interviewees (eight policy makers and eight of the journalists) say that they feel the need for an organisation like H&P is probably or definitely increasing. None say that it is decreasing. The reasons for this are around loss of institutional memory because of the high staff turnover in government and the move away from paper based systems.
All of the journalists and eight of policy makers gave the most positive response to a question about their interest in history, saying they had 'a lot' of interest in history.
Journalists and policy makers have a high engagement with H&P's written resources
Nine (out of 10) journalists and nine (out of 10) policy makers have read an H&P publication. Written resources are more regularly accessed than social media. Five (out of 10) journalists and seven (out of ten) policy makers say that H&P resources are of a high quality.
Journalists and policy makers have a relatively high view of H&P's understanding of their area. Five (out of 10) policy makers feel that H&P is 'strong' at understanding policy making (and the other five answered 'neither' or don't know). Four (out of 10) policy makers feel that H&P is 'strong' at responding to policy makers. Five (out of 10) journalists feel that H&P is 'strong' at understanding the media (and the other five answered 'neither or don't know').
Six (out of 10) journalists feel that H&P is 'very strong' or 'strong' at responding to the media (the other four answered 'neither or don't know'). The number of 'don't knows' indicates that H&P could strengthen its relationship with stakeholders.
Respondents generally report few barriers to accessing H&P's resources. The main barrier, reported by five respondents, is 'information at the wrong time'.
Impact on policy tends to be indirect rather than direct
Given the complex and multifaceted nature of policy, interviewees were naturally reluctant to link specific policy directions to H&P. It seems that H&P's impact is more indirect, in changing the way stakeholders think, than in translating specific lessons from the past into practice.
17 out of 20 respondents say that H&P has reinforced their interest in a historical perspective on their work. Just over half of interviewees state that H&P has given them contacts, namely historians, to build a historical perspective within their work. Six journalists say that H&P has given them a story and six policy makers say that H&P has given them insights into how policies might work in practice. One policy maker says that engagement with H&P has given them a new policy idea.
H&P can be more influential
Respondents gave clear guidance on how H&P can increase its relevance to them:
- Information could be more accessible and vivid in language.
- Information could be more practical and precise in translating the implications from history into action.
- Information could be more targeted in its content and timing, taking account of online and live debates from think tanks and other vehicles.
- H&P should have more face-to-face meetings to establish relationships with journalists and policy makers.
Following the the completion of both phases of the evaluation, H&P held a policy seminar in King's College London to discuss these issues and how historians could best make an impact on the policy debate. Both the agenda [PDF, 202 kB] and the synopsis [PDF, 106 kB] from the seminar are available to read.
Support for the evaluation
We are very grateful to the Esmée Fairbairn Foundation for supporting this evaluation.