On a sweltering day in June 2022, 50 knowledge exchange activators, curators, managers and researchers met to consider ways of measuring, valuing and reporting the impacts and outputs of cultural knowledge exchange. This informal, peer to peer discussion was held against the backdrop of recent reviews of the Knowledge Exchange Framework (KEF), the HE-BCI Survey, and an ongoing Review of Knowledge Exchange Funding being undertaken by Research England. Many of us working in KE have spent a considerable amount of time in recent weeks and months thinking about and discussing knowledge exchange metrics, but the need for a more creative and wide ranging discussion about how to articulate the impact of cultural knowledge exchange was keenly welcomed. Conversations focused on potential models from other disciplines and sectors that might help better capture some of the rich, complex yet often less tangible benefits of work of this nature, and also how our KE partners and collaborators could be better represented, rewarded and platformed within current reporting structures.
Whilst there was a considerable amount of pragmatism in the group around the fact that Metrics are of course here to stay, and that they are a necessary and beneficial tool for funders, there was wide agreement that what is measured, and how it is captured should be considered as the relatively nascent field of cultural knowledge exchange matures.
There was concern that the quantitative metrics currently focus HEIs on delivering outputs, rather than outcomes (for instance the numbers of people/companies supported or partnered with, rather than the change that might have been made through the partnership). This approach could restrict the ambition of HEIs, and certainly limits the ability of institutions to communicate the full value of projects.
Monetary metrics are not reflective of all the benefits that KE with the Arts, Humanities and Social Sciences (HASS) elicits, and the benefits of a mixed methods approach to conveying the richness and complexity of the work, alongside existing tools, was widely acknowledged. Part of the complexity is of course the huge range of work which comes under the banner of “knowledge exchange” and the challenges that this would present for the sector in the adoption of any one framework. Whilst the approach taken within REF to Impact Case Studies was admired for the rigour, detail and utility of the studies, there was an understandable nervousness about creating a “REF 2” within KEF, because of the enormous burden on staff. It was mooted that perhaps KEF case studies could be limited to those disciplines and projects that would benefit from a more nuanced and narrative approach, where there were no existing metrics that effectively captured KE in that particular project or discipline. It was also felt that the REF style approach of focussing on processes (i.e. how was value created, and what is the value beyond proxy measures) in a selected number of case studies could be effective and less onerous.
As one would expect in a group concerned with cultural knowledge exchange, the issue of income as proxy was the subject of much discussion. In the strange policy environment we are currently operating in, it was felt that although financial metrics are certainly here to stay, in tandem with the “shelving” of the industrial strategy and increased emphasis on the levelling up and civic agenda, there were perhaps opportunities to think differently about proxy, and focus on developing proxies that might look at the value out from a university to partners, collaborators and communities, rather than income in to the university from KE activity. When considering “inward” proxies, it was felt that there was also scope in measuring more than finance to better articulate the gains that KE brings to an institution; new knowledge and expertise, new avenues for research enquiry and positive ecologies that nurture and support academic excellence in teaching, research and expertise. Proxies around these inward benefits would also assist with articulating and better valuing partner contributions to KE; which would help an emphasis on the all-important exchange that's central to KE within reporting.
Of course, the development of these “alternative proxies” is quite complex. As we know methods for the measurement of social value (e.g. the government’s Social Investment Bank) are often adopted in sectors such as housing or social care, and many HEIs are working on their own frameworks and systems of measurement (potential for collaboration and sharing here?). These methods are also frequently discussed and sometimes utilised by funders such as the DCMS or Arts Council England, and although they do offer solutions in terms of articulating complex and diverse impacts, they involve a significant amount of effort and expense to operate effectively. Longitudinal impact tracking is often key to capturing these impacts, and that again is costly and resource intensive. The acceptance that some projects will “fail” and indeed need to be allowed to fail well and quickly without the institution being penalised was also discussed, as was the danger that without a longer term approach metrics could stifle the development of a potentially significant project if early measurement shows only modest success.
Discussion suggested that there were models used in social sciences, perhaps not widely adopted in the arts and humanities or cultural knowledge exchange that could be effective in developing alternative proxies, for example methods employed in ethnographic evaluation, or even that the methods the DCMS uses to measure social value in capital development could be adapted to project use. Anecdotally we heard of models in science and technology that really championed the voice of the user/partner and captured wider impacts of the work. However, these models were generally applied when working in collaboration with wealthy or very well funded partners, who contributed a significant amount of resources to the project.
The question therefore remains - how do we authentically platform the voices of cultural or community partners without creating a drain on their already limited resources. And if/when we can achieve a method for doing so, how do we make the feedback really count, if it cannot be quantified as a metric. This feels like a fundamental step if we are to effectively measure the value of HE’s work in the cultural sector.
Moving forward: a call to action
We are delighted to have received the following responses to this blog and the issues that it raises. If you would like to add your own thoughts, please contact Suzie Leighton suzie@tcce.co.uk to discuss your ideas further.
Response 1 - Measuring Cultural Knowledge Exchange’s Real Value: Five of my Favourite Assumptions
Response 2 - Measuring ‘real value of our Cultural Knowledge Exchange work’ UKCEH thoughts
Authors
Suzie Leighton, Co-Director NCACE / TCCE
Dr Astrid Breel, Impact Research Fellow, Bath Spa University
Sian Brittain, Head of Innovation, Guildhall School of Music and Drama
Diagram: DCMS cultural value framework