Evaluation has been an integral part of the NCACE project from its early planning stages. The key purpose is to ensure ongoing dialogue and connectivity across the NCACE Areas of Work to enable us to have a clear understanding of what is working, or indeed not, and how we need to respond to that across each strand. The aim is to add value by enabling the ongoing evaluation of both the NCACE results or goals (i.e. outputs, outcomes and longer-term outcomes or impacts), as well as the processes used to achieve them. Ultimately, this will ensure effectiveness and efficiency in the co-design and delivery of the NCACE activities. In addition, this approach supports flexibility and adaptability as appropriate. The evaluation of NCACE is guided by relevant principles, including logic frameworks and Theory of Change, while stakeholders across the programme further inform the co-creation of a dynamic evaluation framework. The integration of a prescriptive (e.g. specific NCACE mission and clear goals) as well as an emergent (e.g. open-ended and flexible) approach, results in a hybrid evaluation strategy for NCACE. This allows not only for maximum utilisation of available resources, expertise, skills and abilities, but is also a catalyst for more tailored or innovative approaches to evaluation (e.g. via ongoing co-creation of evaluation tools with key project stakeholders).
As we are approaching the completion of the first eight months of the project, it has already become evident that evaluation is a valuable reflection and learning tool for the NCACE core team. The formative NCACE evaluation is guided by Theory of Change (ToC), which reflects the connections between different levels of NCACE goals. These levels of goals include the short-term, intermediate, and longer-term goals (results or outcomes) pursued by NCACE. Through a series of evaluation workshops, the core NCACE team co-developed five ToC causal frameworks, one for each NCACE Work Package (WP). This process involved a number of key steps. First, the process began by identifying or clarifying the long-term goal of each WP. This was critical in allowing the core team to clearly articulate the goals that each WP was set to achieve in alignment with the overall NCACE strategic purpose. Second, once the long-term goals were identified, working backward, the causal linkages between the long-term goals, intermediate outcomes and outputs were established. During this process, the core team engaged in in-depth discussions about what were the preconditions or prerequisites for each goal on the ToC causal framework to be achieved. In addition, the process allowed the core team to determine the underlying assumptions in each key causal link, as well as potential activities (or interventions) that need to take place for each goal to be achieved. Third, the key resources necessary to perform all the activities and interventions were identified, reflecting on capacity or resource availability issues. Fourth, KPIs were set for each key NCACE goal.
Co-designing the ToC frameworks for each WP has been an iterative process and was particularly messy at the start. However, it helped the core team articulate the NCACE results more clearly, identify areas of strengths and weaknesses, discuss alternative solutions, justify the choices made and set priorities. Moreover, it allowed for the core team to determine goals that reflect the NCACE purpose and set KPIs that will be used as measures to evaluate the level of NCACE success. Overall, the process of developing the ToCs opens up creative thinking, strengthens NCACE strategies and overall effectiveness, providing a solid framework for monitoring and evaluation, while supporting the description of the overall impact the NCACE will make or hopes to make.
Facilitating the co-development of a ToC for each WP has been a challenging but very rewarding task. The expertise of the core team in terms of how change happens in the arts and cultural sectors and in HEIs, combined with the evaluation expertise brought by me, as the NCACE Evaluation Lead, is proving fruitful. We have been utilising a combination of ToC diagrams and ToC descriptive narratives to facilitate communication of NCACE evaluation ideas within the core team. Some of the challenges faced in the process of co-developing a ToC for each WP have, however, stemmed from the ToC’s technicalities, the use of specific evaluation terminology, as well as the scope and complexities of the specific project. The process, expectedly, has been somewhat demanding in terms of the core team’s time and energy and sometimes felt a bit unyielding. To a certain extent, this is anticipated as evaluation is an iterative process and the NCACE project is quite a substantial one.
It is clear that the process of co-developing a ToC for each WP of the NCACE project has been beneficial. The process supports a systematic critical thinking about what value NCACE intends to create. Importantly, it helps to organise ideas, examine assumptions, prioritise activities and tackle the complexities of the NCACE project. Therefore, the process provides a roadmap of how NCACE will reach its desired results or goals. Furthermore, the developed ToC frameworks will serve as a basis for stakeholder communications and buy-in. Overall, ToC provides a framework for implementation, collaboration, monitoring and evaluation for NCACE. It should be noted that the ToCs developed for each WP are not static. Reflecting the dynamic contextual environment within which they are formulated, they evolve through an ongoing process of discussion-based analysis and learning that produces powerful insights to support project design, strategy, implementation and evaluation.
Image: NCACE Work Package (WP) 1 Theory of Change Framework (Credit: Dr Thanasis Spyriadis)