9.2 State of the art

Measuring progress, effect and impact on transition issues is essential to demonstrate whether the predetermined goals have been achieved, and whether this can be traced back to the interventions that have been carried out. This not only concerns the direct effects (output), but also which expected and unexpected changes take place (outcome), why they have changed, and what the impact of these changes are on the systems (impact). While monitoring progress, information can be systematically and / or continuously collected and analysed. Because monitoring takes place during the project, it is possible to make timely adjustments if things do not go according to plan. Effect measurements complement monitoring activities and provide in-depth and objective insights into the relevance, efficiency, effectiveness, impact and sustainability of the intervention at specific times.

The use of the right KEMs depends, among other things, on the intended goal and the context in which the transition takes place. Interventions with a clearly specified end goal require a different M&E method than interventions that focus on structural change where the desired end goal is (still) unknown. This also applies to the environment in which the change takes place. Changes in complex and dynamic systems cannot be monitored with methods that are based only on and depend on protocoled data, structure, and certainty. This also requires new methods. For this chapter, the M&E methods are described on the basis of a conceptual framework that describes the dynamics of monitoring and adjustment. In this context, we distinguish methods aimed at targeted M&E and methods aimed at learning M&E. Targeted M&E methods often start with predetermined goals, where progress is monitored via measurable indicators that are selected at the start. The strength of these methods lies mainly in justifying the project goals and demonstrating relationships between activities within the intervention and the results. Learning monitoring methods can better deal with uncertainty about the approach in advance and the unexpected effects during the process, whereby the strategy can be adapted to the changes during the intervention.

Targeted M&E methods Methods based on targeted M&E are mainly used to justify projects and interventions. By setting the goal before the start of the project, indicators are chosen that can demonstrate whether the goals are being achieved. It is therefore important to have a clear and unambiguous picture of the expected effects of the intervention in advance. These methods are highly regarded in science. However, these methods are often not suitable for interim assessment of assumptions and adjustment during the process. The randomised controlled trial (RCT) is an example of a project evaluation method that provides insights into the direct relationship between activity and outcome, because the effects of the intervention are compared with the effects in a comparable population (the control group) without intervention (Donaldson et al. al., 2015). However, experiments with a randomised and controlled design are often time consuming, cumbersome and static. Small-scale experiments with randomisation - as applied in Rapid Cycle Experiments - can quickly gain insight into which parts of an intervention work, in order to further develop and optimise the intervention on that basis (Johnson et al., 2015).

On the basis of a Social Cost-Benefit Analysis (SCBA) an accurate estimate can be made in advance of the expected effects of the intervention. This method maps the positive and negative effects and is therefore used to justify policy measures. The method focuses on the welfare effects of the measures, and in addition to the economic effects, an estimate can also be made of the so-called soft effects, such as the impact on culture, happiness and well-being. SCBAs place high demands on the quality of information and research methods that are used as input, making this method of limited use for unstructured and incomplete datasets and projects with uncertain outcomes. Certainly within transition issues of a complex nature, these are aspects that surface more often (Koopmans et al., 2016). After the intervention, variants of the SCBA can be used as evaluation measurements, such as the cost-effectiveness analysis. Surveys and the registration of indicators provide a good picture of improvements and changes after the intervention. An example of this is the Care Monitor, which provides insight into the performance of health care based on a broad set of predefined indicators (van den Berg et al., 2011).

Learning M&E methods Transition challenges often concern complex changes in systems in which both the required approach and the expected effects of this approach are difficult to estimate in advance. Transition issues therefore often require an M&E method that is dynamic and adaptive. Within various disciplines, methods have been developed that are in line with the uncertain nature of transition issues and that move with the changes in the transition. For example, the popular Agile working method has in recent decades spread from software development to industry, and is now increasingly emerging in digital and non-digital projects in science. Recent developments from data science in the field of AI and big data also make new methods available for M&E. With the help of data-driven predictive analyses, real-time insight can be generated on the effects of the interventions. More about the opportunities and challenges of these methods is described in the section with challenges and research questions.

Within the behavioural sciences we see methods emerging that attempt to combine dynamic monitoring with scientific justification that we know from randomised trials. For example, N-of-1 studies (or Single Case Design) can monitor the direct effects of interventions on behaviour, based on repeated quantitative measurements within an individual over time (McDonald et al., 2017). An important advantage is that the intervention can be further developed and adjusted during the measurements. Another advantage is that the baseline can be different for each participant. This gives you insight into individual differences, the effect of the context, and you can minimise the statistical disadvantages of distribution in the target group. However, N-of-1 studies are mainly suitable for digital behavioural interventions, and are very dependent on the intervention adherence of the participants. Still, this is a promising method for transition issues with a behavioural change component. Other promising methods that identify effective mechanisms in design propositions and interventions are MOST (Multiphase Optimisation Strategy; Collins et al., 2007) and CIMO-logic (Denyer et al., 2008).

Reflexive Monitoring in Action (RMA) is a participatory M&E method developed to monitor the progress of system innovations (Van Mierlo et al., 2010). It facilitates the development of learning processes during transitions and thus stimulates the determination of the direction of the transition. The determination of the goal, approach and indicators is moving along with the progress of the process. Although the specific monitoring tools differ per topic or ambition, it is important that these activities are an integral part of the transition. Examples of methods that can be used in RMA include Theory of Change, Learning History and Most Significant Change Method. The monitoring activities are seen as project activities, where each monitoring cycle consists of the steps ‘observe’, ‘analyse’, ‘reflect’ and ‘adapt activities’. Because reflexive monitoring is an adaptive method, direction can be changed during the project, and unexpected effects can be identified. However, the participatory nature of reflexive monitoring is very important. In order to bring about institutional change, it is essential that all stakeholders participate in this.

Related to reflexive monitoring is the Measure Knowing Act system, developed for the Delta Programme (Loeber & Laws, 2016). Structured reflection moments stimulate ‘learning during the intervention’. It is possible to respond to new developments, to slow down or accelerate activities and to adapt the strategy based on changes in systems. Adjustment takes place on the basis of four main questions: is the project on schedule (budget and time), is the project on track (are goals being achieved), is there an integrated approach, and is there broad participation of stakeholders?