2.8.4. Discussion of findings

We are still at an early stage in understanding how best to adapt to future climate change, how risks can be most effectively reduced and resilience enhanced, and what the characteristics of a well adapting society might be. Therefore, learning what works well (or not), in which circumstances and for what reasons is critical (Pringle, 2011) and this is the reason for doing MRE.

There are two types of learning when considering an adaptation intervention, firstly the learning about the method or approach (linked to the purpose) that is used to measure the progress of the intervention and secondly the learning about what makes ‘successful adaptation’ so that we can move towards creating well adapted societies.

It will be important to share knowledge from MRE schemes into the future

It will be important to continue to share knowledge and results from countries’ MRE schemes at regular intervals into the future. At the moment there are only a limited number of countries (five) who are currently implementing an MRE scheme therefore there is a limited amount of information from which to gather lessons learned. But nevertheless as these countries are ‘earlier initiators’ in terms of adaptation MRE, other countries can learn from their experiences, thus speeding up the process of MRE scheme development for others.

In addition, the fact that nearly half of countries are planning to develop an MRE scheme in the future will mean that there is much more information in the future from which to learn, both about the approaches and successful adaptation. This will provide a good opportunity to share knowledge and learning.

Agreeing on the purpose for the MRE scheme is key

Communicating and agreeing on the purpose for monitoring and evaluating is a key factor when developing an MRE scheme (EC, 2013b) along with acknowledging the tensions and synergies between these different purposes (Spearman and McGray, 2011). The self-assessment survey shows that MRE can be used for a range of purposes thus it is not surprising that it also seems that a variety of approaches are being developed to meet these distinct objectives. This demonstrates that countries are tailoring their approach to meet the purpose of their scheme and recognising that adaptation is context- specific. However, it is recommended that MRE schemes should go beyond considering ‚‘Did we do what we said we would do?’ and explore the unintended and unexpected outputs and outcomes which may stem from a policy or project (Pringle, 2011). Often some of the most valuable adaptation insights and innovations stem from things we didn’t know would happen and these are not revealed if we stick to a rigid examination of performance against predetermined objectives. For example if a city had planned to increase its blue and green infrastructure by creating a park  and water body to reduce urban heat and be used as a flood management intervention. There could be unintended issues related to biodiversity, however these could be both positive – extension of habitat for existing species or, negative - arrival of a new vector-borne disease. This may be where some of the most important adaptation lessons can be learnt.

Since only a few countries are evaluating their implemented adaptation measures there is still debate about ‘what successful adaptation looks like’. Thus European countries should not necessarily have common indicators to measure success but should strive for a common understanding of success and also what contributes to failures (Chan, 2013). The UK example goes beyond reviewing specific objectives and demonstrates a broader purpose for the evaluation. The ASC (ASC, 2011) applies its adaptation framework (“adaptation preparedness ladder”) to answer three key questions in the context of a number of key themes or sectors:

1. Is the UK becoming more or less vulnerable to risks from current and future climate?
2. Are we seeing sufficient uptake of low-regret adaptation actions?
3. Are long-term decisions systematically accounting for climate risks?

Answering these questions could therefore be considered as the purpose of the approach.

The importance of stakeholder involvement

According to the EC Guidance (EC, 2013b) ‘Engaging and involving affected stakeholders’ is a key factor when developing an MRE scheme. Stakeholders can range from the agencies that set the objectives to the parties that are the intended recipients of the intervention. They also include the intermediary organisations that are involved in implementing the intervention, such as related ministries, sub-national government, sector bodies, expert institutions and the media.

The self-assessment survey shows that stakeholder involvement in the monitoring and evaluation stage of the policy planning process has so far been limited. Where countries are already implementing their MRE scheme such as Finland, Austria and the UK the examples show that stakeholders were actively involved in the process, often through workshops to enable two-ways discussion. Engaging a wide range of stakeholders means that everyone has a ‘voice’ and there is a greater chance of identifying an intervention that may be a benefit for one group but is maladaptive (maybe harmful) for another. The risk of maladaptation (EEA, 2013) can be reduced by using MRE for learning, reflection, and improvement of ever-evolving strategies.

The benefits of looking both "top-down" and "bottom-up"

National strategies and implemented actions would be improved if they took into account learning from ‘top-down’ sources i.e. relevant international and national sources that may provide examples of methods or measures. They also need to take account of ‘bottom-up’ sources e.g. local, regional and sectoral sources of information that may provide knowledge of how social and cultural beliefs influence how particular groups respond to measures. Lessons from international reviews of MRE have concluded that it is useful to look both ‘top-down‘ and ‘bottom-up‘ (Chan, 2013) and countries have demonstrated that they are doing this. Finland, Austria, the UK and Ireland have mentioned that they have plans to include learning from vertical and horizontal actions in their national strategy and this also ensures that the challenge of multiple scales and sectors for adaptation is addressed.

The role of institutions – governance, capacity and communications

It is important to be aware of which agency is carrying out the MRE and how their perspective influences the objectives of the scheme. The self-assessment survey demonstrates that different agencies are responsible for MRE in different countries and this can influence communication, institutional capacity and the objectives of the MRE scheme. The examples demonstrate that in Austria and Finland it is a national government Ministry that has responsibility for MRE and in the UK and Germany there is an independent coordinating body. Setting up an independent coordinating body can strengthen communication between relevant institutions, improve synergies, help to identify best practice and lessons learned and make the process transparent (Olivier et.al., 2013).

The GIZ discussion paper (Leitner, 2013) states that the information from the MRE scheme not only needs to be used in a way that feeds into the relevant decision-making process, but is also presented in the right way for the target audience (typically different levels of policy-makers). Therefore the strength of communications between institutions, as well as the way that information from the MRE scheme is presented to those who need to use it is important for enhancing learning opportunities and building institutional capacity.

Challenges to address when developing an MRE scheme

As discussed earlier there are a number of challenges relating to MRE for adaptation (Bours et. al., 2014) and this section considers some of these challenges and reviews how countries are dealing with them.

Adaptation is a continuous process

Adaptation is a process of continual adjustment which, if successful, will enable socio-economic or environmental goals to be achieved despite a changing climate context. The self-assessment survey shows that countries are planning to use the information from their monitoring and evaluation schemes to revise either their national strategy or action plan. It also shows that France and Finland are carrying out both a mid-term and end of programme review and using the results to feed into a new strategy. It is promising that countries have recognised that adaptation is an iterative process whereby learning from MRE schemes and new information from research are fed back into the process to improve the adaptation intervention. This is also a recommendation by the UNFCCC (2010).

Adaptation cuts across scales and sectors

Climate change is global but adaptation is local. Therefore adaptation interventions should be context-specific reflecting national approaches and prioritising local knowledge and circumstances. The Finish and UK examples demonstrate that the approaches being developed aim to consider evidence across both sectors (horizontal) and scales (vertical). They also consider interdependencies, cross-cutting issues and unexpected issues.

The linked issue to this is that the range of different actions (at different scales) makes it difficult to compare or aggregate results. The desire to aggregate results can lead to an over-dependence on quantifiable indicators which due to the reasons listed below should be cautioned against.

Uncertainties are inherent when dealing with adaptation interventions

As discussed earlier uncertainty about the scale, timing and spatial nature of how the climate might change (ASC 2011) and how society might respond makes it challenging to define good adaptation. MRE schemes need to acknowledge these many uncertainties and design in appropriate baselines that track contextual changes and flexibility in the approach and indicators.

The UK and the Austrian examples show that the approach chosen is flexible and iterative allowing the intervention to adjust over the long time frames as circumstances change. Another advantage of flexibility in the approach is to avoid lock-in to potentially mal-adaptive measures.

Measuring progress using indicators

The measurement of progress using indicators for an adaptation intervention is more challenging than for many other fields, including mitigation, because of the long time frames before the outcomes of the adaptation intervention are known, the lack of data, the complexities of generating baselines, uncertainties and attributing the results to the adaptation actions taken (Mullan et. al., 2013).

The self-assessment survey shows that ten countries have already developed or are developing indicators. Austria has used ‘process-based’ indicators that allow them to determine whether progress is on track even if outcomes cannot be determined yet. Some countries e.g. Germany (EEA, 2013 p90) have both ‘impact indicators‘ and ‘adaptation response indicators‘. Progress indicators that focus on activities are also used for international reporting such as National Communications for UNFCCC and Climate Adapt. The use of a combination of quantitative, qualitative and ‘binary’ indicators that forms a context-specific suite is supported by an OECD paper. It also recommends that the qualitative indicators are developed and supported by direct dialogue with beneficiaries such as focus groups. (Lamhauge et.al., 2011)

In Germany and Austria the indicators are mainly based on existing M&E systems and this is reinforced by the EC Guidance (2013b). The guidance recommends: not reinventing the wheel- since in some cases existing M&E systems that are already used in other fields can be adjusted to better account for adaptation. Among the many challenges when developing indicators is the availability of data. The data used for quantitative indicators ideally needs to have national coverage, be collected on a regular basis, statistically validated and publicly available. To meet these rigorous standards can be cost and time consuming so using existing data is the logical next step. Hence in Austria (section 2.8.3) and Germany (Leitner, 2013) the indicators are primarily based on existing data sources.

Lastly the Austrian, UK, Lithuanian German and Finish examples show that there is variation in both the indicators and the approach. This is positive and reflected in the guidance in that it recognises that schemes and co-ordinated indicators should not all be the same but tailored to national circumstances.