A Methodology for Ex-Post Evaluation of Local Climate Change Mitigation Actions under a Multi-Level Governance Framework

(1)

A METHODOLOGY FOR EX-POST EVALUATION OF

LOCAL CLIMATE CHANGE MITIGATION ACTIONS

UNDER A MULTI-LEVEL GOVERNANCE FRAMEWORK

Isabel Azevedo

Doctoral Program in Sustainable Energy Systems Faculty of Engineering of UP

2019 Supervisor

(2)

(3)

Isabel Maria Gonçalves Azevedo

A Methodology for Ex-Post Evaluation of

Local Climate Change Mitigation Actions

under a Multi-Level Governance Framework

Supervisor: Prof. Doutor Vítor M. S. Leal

Thesis submitted to the Faculty of Engineering of the University of Porto in partial fulfilment of the requirements for

the degree of Doctor in Engineering

Department of Mechanical Engineering Faculty of Engineering of the University of Porto

(4)

(5)

This thesis was developed in the following institution:

(6)

(7)

To my parents, my sister and Pedro.

(8)

(9)

Acknowledgments

Throughout this four-year journey, I was fortunate to have had the contribution and support of several people. This section is a sincere thank you to all of them.

I would like to take this opportunity to thank my supervisor, Professor Vítor Leal, for sharing his knowledge and time over the last years. I have to thank him for the motivation and guidance that led me to where I am today; and also for his insights and for the discussions that were crucial for the development of this work.

I would also like to acknowledge the financial support from FCT – Fundação para a Ciência e a Tecnologia, under the MIT Portugal Program (PD/BD/105862/2014); and the Energy unit at INEGI – Institute of Science and Innovation in Mechanical Engineering, for hosting me.

This is also the opportunity to recognize the importance of all the people and institutions that helped me with the collection and sorting of the data used in this study. I am thankful to AdEPorto - Agência de Energia do Porto, for granting me access to the data from the energy matrix of Porto and to the information concerning local actions towards climate change mitigation (studies and projections as well as monitoring data). A special thank you goes to Engo _{Alexandre Varela for the effort necessary to}

provide me with the required data and for his availability to discuss any issues regarding local level data. I would also like to acknowledge ADENE - Agência Portuguesa de Energia, for making available the energy certification database used in this work, and the Statistiska centralbyrån (SCB), for their support in finding the appropriate data sources for Swedish statistical data. I am also grateful to the municipality of Porto, namely to the Divisão Municipal de Conservação, Infraestruturas e Gestão de Frota, for granting me access to detailed information on the energy consumption for the municipal fleet.

At FEUP, I would like to thank to Professor Isabel Horta for sharing her knowledge in statistical analysis and causality tests. I am also grateful to my colleagues for their help and discussions. A special thank you goes to Mafalda for the daily support and all the good conversations.

During this four-year period, the care and companionship of my friends was essential. Therefore, I want to thank all those who were there (even when being geographically distant) to support and accompany me through this journey.

I am deeply grateful to my family for their immense support and patience. To my parents for believing in me and for making me who I am today, for their unconditional love and support. To my sister, with whom I can always count, for listening and encouraging me. I am profoundly thankful for being her sister. To my grandparents, who I just want to make proud. To the rest of my family.

Last but not least, a very special thank you goes to Pedro. For sharing this journey with me, for his patience and support, in the good and in the bad moments, for all the laughs. Thank you for everything.

(10)

(11)

Resumo

A importância do papel das ações locais de gestão de energia para o cumprimento dos objetivos globais de mitigação das alterações climáticas é globalmente reconhecida, tanto pelo meio académico como pelos diferentes decisores políticos. Caracterizar o seu impacto torna-se assim essencial. A avaliação das ações locais é crítica para o desenvolvimento de novas políticas assim como para a identificação e promoção de boas práticas. No entanto, a apreciação das alterações do sistema local de energia durante um certo intervalo de tempo não é suficiente para a avaliação dos efeitos associados às ações locais. O que é observável não corresponde ao impacto específico das ações locais, uma vez que existem outros fatores - não totalmente controláveis pelas autoridades locais (fatores externos) - que provocam alterações ao sistema energético local. Esta tese foca-se então nesta problemática, na necessidade de avaliar de forma sistemática as ações locais e isolar o seu impacto tendo em consideração os fatores externos e os respetivos impactos no sistema local de energia.

Sendo a avaliação de políticas energéticas e climáticas um tema tratado por diversos estudos (científicos e não só), foram já desenvolvidas e apresentadas diferentes abordagens metodológicas de avaliação que podem, de certa forma, contribuir para o objetivo acima proposto. Contudo, nenhuma das abordagens existentes resulta na avaliação da contribuição real das ações locais para a mitigação das alterações climáticas. O repto refere-se essencialmente à possibilidade de desagregar os efeitos associados às ações locais dos efeitos associados aos fatores externos.

Tendo como principal objetivo a avaliação da real contribuição das ações locais, foi necessário proceder à identificação e caracterização de todos os fatores com impacto relevante no sistema local de energia. Este processo foi realizado através de uma revisão sistemática da literatura existente, resultando na identificação de quatro grupos distintos de fatores – o contexto local, a evolução natural do sistema energético, o contexto de governança nacional e internacional vigente, e as ações locais para a mitigação das alterações climáticas. Nesta tese é proposta uma metodologia que, tendo por base os diferentes fatores e a sua relação com o sistema local de energia, desagrega as mudanças do sistema de energia nos efeitos individuais associados a cada fator. O efeito real das ações locais para as alterações climáticas é assumido como sendo o “resto” entre as mudanças observadas e a soma de todos os efeitos associados aos fatores externos. A metodologia proposta é aplicável a diferentes contextos, e a informação necessária para a sua implementação corresponde a dados que estão comumente disponíveis.

Finalmente, de forma a validar e refinar a metodologia proposta, esta foi aplicada a dois casos de estudos, Porto e Malmö. Em ambos os casos, os resultados obtidos são robustos, existindo coerência entre os efeitos estimados e uma análise bottom-up das ações implementadas. Foi ainda efetuada uma análise comparativa entre os resultados obtidos e os objetivos estabelecidos pelas autoridades locais. Esta análise permitiu, não só avaliar o progresso dos municípios face aos seus objetivos, como reforçou a ideia de que a mera apreciação das mudanças observadas pode ser bastante enganadora no que diz respeito à contribuição das ações locais para a mitigação das alterações climáticas.

(12)

(13)

Abstract

Local actions on energy are currently seen as of major importance for the achievement of climate change mitigation overall targets. It is therefore essential to be able to assess their influence. The assessment of local actions is important as it produces knowledge that adds value to the development of new policies and the promotion of best practices. However, the effects associated with local climate change mitigation actions cannot be properly assessed by merely looking at the changes that have occurred in the local energy system within a specific timeframe. The observable changes, as a whole, do not correspond to the local actions’ impact on the local energy system, as they are also caused by factors that are not fully controlled by local authorities (external factors). This thesis addresses this issue, focusing on the development of an evaluation methodology that takes into account the effects associated to the external factors of change in order to isolate the effect of the local actions.

The evaluation of climate and energy related policies has been a subject of research by both academics and policy makers. There are several studies that contribute to the assessment of local climate change mitigation actions by presenting different evaluation approaches. Despite the diversity, none of the existing approaches leads to an evaluation of the specific contribution of local actions towards climate change mitigation. Thus, the main challenge of this work is to decouple the effects associated to local actions from the effects associated to the remaining factors of change of the local energy system.

The first step towards the evaluation of the actual contribution of local actions towards climate change mitigation corresponds to the identification of all the relevant factors that impact the local energy system. Existing literature on the different factors that lead to changes in the local energy system were reviewed in this thesis, resulting on the identification and characterization of a set of factors of change that can be combined in four groups: local context; natural evolution of local energy system; higher-level governance framework; and local climate change mitigation actions. Based on the identification and characterization of the different factors of change, a methodology that allows for the disaggregation of the observed changes is developed in this thesis. It is assumed that local actions effects are the “remaining gap” between observed changes and the sum of all the effects from external factors of change. The proposed methodology is applicable to different contexts and uses as inputs data that is commonly available.

The methodology was applied to two case-studies, Porto and Malmö, in order to validate and finetune of the model as well as to demonstrate how it can be made operational. The results obtained for the two case-studies were robust, i.e. the estimated effects of local climate change mitigation actions were plausible and congruent with the bottom-up analysis that was also performed for both cases. Finally, this thesis also includes a comparison between the estimated effects of local actions with the projected outcomes (established ex-ante by the two municipalities). This comparison provides an understanding on the municipalities’ progress towards the established goals, and reinforces the fact that the mere assessment of the observed changes can be strikingly misleading.

(14)

(15)

List of abbreviations

• ADENE, Agência de Energia

• AdEPorto, Agência de Energia do Porto • Agri, Agriculture and Fisheries sector • BY, Base Year

• CC, Climate Change • CDD, Cooling Degree Days

• CO2, Carbon Dioxide

• CoM, Covenant of Mayors for Climate and En-ergy

• DECC, Department of Energy and Climate Change (United Kingdom)

• DGEG, Direção Geral de Energia e Geologia (Portugal)

• EPBD, Energy Performance Buildings Directive • EU, European Union

• EU-ETS, European Union Emissions Trading Scheme

• FE, Final Energy • GHG, Greenhouse Gases • HDD, Heating Degree Days

• HVAC, Heat Ventilation and Air Conditioning • Ind, Industry sector

• INE, Instituto Nacional de Estatística (Portugal)

• KLIMP, Klimatinvesteringsprogram

• LEED, Leadership in Energy and Environmental Design

• LIP, Lokala Investeringsprogram • NZEB, Net Zero-Energy Buildings

• ONS, Office for National Statistics (United King-dom)

• PE, Primary Energy • PL, Public Lighting

• PNAEE, Plano Nacional de Ação para a Eficiência Energética

• PT, Portugal • PV, Photovoltaic

• Res, Residential Buildings sector • RQ, Research Question

• RY, Reporting Year

• SCB, Statistiska Centralbyrån (Sweden) • SE, Sweden

• SEAP, Sustainable Energy Action Plan • Ser, Services Buildings sector

• Tr, Transportation sector • UE, Useful Energy • UK, United Kingdom

(18)

(19)

CHAPTER 1

Introduction

The main goal of this work is…

…to contribute for the evaluation of local climate change mitigation actions, identifying the causality between local actions and the respective end impacts. Considering that the evolution of local GHG emissions also depends on external factors (i.e., factors that are not fully controlled by local authorities), the development of a methodology

that is able to decouple the local actions’ effects from the changes prompt by external factors is the focus of this work.

1.1 Context and relevance of the topic

Local actions are currently seen, by both academics and policy makers, as of major importance for the achievement of climate change mitigation overall targets. Both the physical characteristics of local energy systems and the regulatory competences of local authorities prompt local level as an appropriate level for action (Meeus et al., 2011). Indeed, most municipalities have responsibility on planning issues (including land-use planning and management of resources), interfering thus with the main local activities and the use of resources (Dodman, 2009). Municipal authorities are also policy makers, with strong competences regarding buildings (e.g. licensing) or the management of public transport (Khare et al., 2011). Additionally, they are themselves energy users (e.g. public lighting, municipal fleet, and buildings), which gives them the opportunity to lead by example (Kern & Alber, 2008). Finally, municipalities are the administrative level that is closer to citizens, with the best understanding of the local habits and culture, as well as forms of adaptation of the city and its energy services to the population needs.

In the past few years, local authorities and communities have been gradually recognizing this importance of local climate and energy policies (Kern & Alber, 2008). Almost 7000 European municipalities have

(20)

joined the Covenant of Mayors*, despite its voluntary nature. Within and outside the European Union (EU), other initiatives have emerged – ICLEI (2016), CIVITAS (2016), C40 Cities (2016), Cities for Climate Protection (Betsill & Bulkeley, 2004)) – promoting and supporting policy action at local level. As an illustration of this increase in local activeness, Figure 1.1 shows the evolution over the last ten years of the number of municipalities that have committed to reduce their GHG emissions by at least 20% by 2020, within the Covenant of Mayors.

Alongside this increasingly active role of local actors, the academic community as well as policy makers have been developing extensive work regarding local energy planning (see Neves (2012) for a review) and the relevance of local actions towards climate change mitigation.

0 1000 2000 3000 4000 5000 6000 7000 8000 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 # Signatories # SEAPs # Progress Reports

Figure 1.1 – Evolution of the adhesion to the Covenant of Mayors between 2008 and 2018, in terms of number of signatories, number of sustainable energy actions plans (SEAPs) approved and number of progress reports delivered (cumulative)

As local actions become more and more patent, it is essential to assess the extent of their influence. Are local actions actually contributing to climate change mitigation? What are their effects? The assessment of effectiveness, relevance and efficiency of local actions is important in the sense that it produces knowledge that adds value for the development of new policies and the promotion of best practices. As said by Vaz et al. (2001), “Evaluative information can usefully serve several purposes, including allowing the assessment of future impacts of environmental policy measures, and informing choices between alternative policy options”. Moreover, this issue is extremely timely, as several municipalities are currently undergoing or have already finished the implementation process of the first stages of their climate change mitigation strategies. For instance, as shown in Figure 1.1, most municipalities that signed the Covenant of Mayors have already developed and validated their SEAPs and are going through their implementation process (CoM, 2016b). Also, about one fourth of the signatories have already published the respective progress reports. Having this into account, it is very important for policy makers to know whether implemented and ongoing actions are fulfilling the objectives for which they were designed and, if not, what are the limitations. The evaluation of climate and environmental policy has been a subject of inquiry by different stakeholders – from academics to policy makers – at several policy levels. Both scholars and practitioners have

*This initiative consists on a voluntary agreement where city authorities commit to reduce their greenhouse gases (GHG) emissions by more than 20% by 2020, compared to 1990 levels (CoM, 2016a). As part of the commitment, city authorities are required to develop a Sustainable Energy Action Plan (SEAP), where they present how they intend to achieve the emissions reduction, and to comply with reporting rules.

(21)

1.2 Previous work: important achievements and main gaps 3

recognized the difficulty in properly assessing the effects of climate related policies, given its universal scope and the large interaction with other policies. The obvious relation between economic growth and energy use is just one example. In the specific case of actions and policies implemented at the local level, these difficulties are amplified by the fact that they are implemented as part of a complex multi-level governance framework, which also includes policies implemented at regional, national and international level. The local energy system is indeed part of a much larger system, being influenced by several external factors – being local characteristics and/or higher-level policies (Figure 1.2).

Higher-level governance framework Local Authority Laws Organizational structure Operating arrangements Local Context Socioeconomic features Assets (Financial, human, physical, natural, social) Vulnerability context (Trends, culture) Strategic decisions / actions on climate change response Strategies individuals adopt Mitigation outcomes Adaptation outcomes

Higher-level governance framework

Figure 1.2 – Schematic representation of the influence of external factors on the impact of local Climate Change actions

Thus, the effects associated with local climate change mitigation actions cannot be properly assessed by only looking at the changes that have occurred in the local energy system within a specific timeframe. The observed changes, as a whole, do not correspond to the local actions’ impact on the local energy system as they also depend on factors that are not fully controlled by local authorities. For example, the effect (in terms of primary energy and GHG emissions) of an action designed to save a certain amount of electricity per year will depend on how the national electricity production evolves concomitantly.

1.2 Previous work: important achievements and main gaps

The literature on the evaluation of climate and environmental policy includes studies performed by different stakeholders – from academics to policy makers – at several policy levels. Relevant work has been done regarding the theoretical principals of policy evaluation and on the quantification and evaluation of policy actions effects. A detailed review of these studies is presented in chapter 2.

The existing studies focusing on the theoretical principals of policy evaluation are important in the sense that they contribute to the identification of the critical features to be included on an evaluation methodology in order to guarantee an adequate policy assessment. An evaluation methodology can be assessed based on how it deals with three types of questions – descriptive, causal and normative. The descriptive refers to questions such as “What has happened? Or what will happen?”; the causal refers to the links between the observed changes and the assessed policy; and the normative refers to inquiries as

(22)

“Are the results satisfactory?” The way these questions should be dealt with may differ according to the evaluation goal and the type of policies being assessed. Even if existing studies do not provide specific guidelines for the development of a methodology for the evaluation of local climate change mitigation actions, the framework in itself is extremely useful to assess and compare the different methodological options.

Regarding the quantification and evaluation of policy actions effects, existing studies provide a wide set of methodologies for the assessment of local (and not only) climate change mitigation actions. Overall, their assessment and comparison were extremely important to understand how different methodological op-tions can affect the evaluation outputs. Indeed, the review of existing approaches led to the identification of some methodological characteristics that are considered essential to guarantee the adequate assessment of local climate change mitigation actions. The preference towards ex-post evaluation approaches, and the need for an exhaustive consideration of the causal links between observed changes and the assessed policy actions, are just some examples (see chapter 2 for more details).

Despite being very diverse, the review has shown that none of the existing approaches leads to an evaluation of the specific contribution of local actions towards climate change mitigation, e.g. by decoupling the effects from those due to supra-local level policies. The effects associated with local actions are not properly quantified, neither regarding the individual impact of each action nor for the whole contribution of the different actions being taken at the local level. The main gap on existing approaches corresponds to the deficient analysis of the causal links between observed changes and the actual policies. The consideration of external factors and their impact on policy effects and on the local energy system in general is seen as the major shortcoming.

1.3 Main contributions intended with this work

This work’s main goal is to fill the existing gap on the identification of the causality between local actions and end impacts, with the development of a new methodology for ex-post evaluation of local climate change mitigation actions. It is intended to quantify the effects associated with local actions by taking into account the external factors, as local specificities and the higher-level policy context.

An analysis and a tool of this kind may not only contribute to inform local authorities on the effects of their own actions and consequently influence future policies but also accelerate the diffusion of good practices and support the development of national and European policies targeting climate change mitigation at local level.

The novelty of the proposed work relies on the combination of three main features: (i) the ex-post evaluation; (ii) the local level as the geographical scope; and (iii) the decoupling of actual impacts of local actions from other effects caused by external factors (i.e. causal links consideration). Even though the existing work on climate policy evaluation already presents some of these features, the review presented in chapter 2 shows that none of them combines these three features, while they are critical for the proper evaluation of local climate change mitigation actions.

(23)

1.4 Research questions 5

regarding the action itself and the observable changes in the local energy system, quantifies the impact of local actions (decoupling the later from other effects caused by external factors).

Alongside, and to facilitate its use by practitioners, the incorporation of the quantification model into a functional tool is also assumed as an important objective of this work. Here, an important issue refers to the balance between the level of detail of the data required and the outputs accuracy. Indeed, the level of detail of the necessary data may be a barrier to the use of complex evaluation methodologies by policy makers. This may be an even more pressing issue in the case of local actors, as their resources for data collection and monitoring are frequently very limited.

1.4 Research questions

Three research questions may be answered in order to attain the aim of the proposed work:

RQ1. What are the external factors that are determinant for the evolution of local energy systems and for the effects of local climate change mitigation actions?

A thorough comprehension of the relevant factors of change of the local energy system is key for the decoupling of the effects associated with local actions from the effects associated with external factors. This includes not only the identification of which factors are determinant in the evolution of local energy systems but also the respective characterization with quantifiable (or qualifiable) indicators.

RQ2. How to model the correlation between the observable changes and the actual impact of local climate change mitigation actions, having into account the relevant external factors?

The answer to this question will allow for the development of a model that determines the causality between local actions and the respective impacts, excluding changes that were caused by external factors. This step is extremely important for the evaluation of local actions, isolating the impact of the assessed actions from changes resulting from factors not fully controlled by local authorities. This decoupling of effects allows for an unbiased assessment of individual actions, being an important step towards the comparison of actions from different municipalities.

RQ3. How different levels of simplification of the proposed model/methodology affect its precision?

Simplification of the evaluation methodology, including the minimization of necessary inputs, could be very important to ensure that its implementation can be done in the future by local authorities or other local actors. However, simplification is usually associated with the loss of accuracy. Thus, it is necessary to assess whether this loss of precision is acceptable given the advantages of the methodology simplification.

1.5 Formal definition of the problem

The current gap regarding the evaluation of local climate change actions stands on two major issues: (1) the identification of the relevant factors of change of local GHG emissions; and (2) the quantification of the

(24)

actual effects. Thus, in order to attain the aim of the proposed work, two distinct, yet complementary, exercises need to be accomplished. The first (T1) refers to the identification and characterization of the different factors of change of local GHG emissions, while the second (T2) corresponds to the development of a model to quantify the actual impacts of local climate change mitigation actions. Figure 1.3 corresponds to a schematic representation of the work performed in order to achieve a proper methodology for ex-post evaluation of local climate change mitigation actions.

METHODOLOGY

(T1) IDENTIFICATION

FACTORS OF CHANGE OF LOCAL GHG EMISSIONS

. Identification of external factors . Characterization of external factors . Characterization of local actions

(T2) QUANTIFICATION

OF LOCAL CC MITIGATION ACTIONS EFFECTS

. Modeling the relation between observed changes and local CC mitigation effects

. Build a tool to perform quantification

(T3) CASE-STUDY(IES) . Application to case-study(ies) for validation of the model

. Fine-tuning of the model

. Reality-check and validation of the results

(T4) CONCLUSIONS . Regarding data collection and evaluation . Regarding the effects associated to external factors . Regarding the effects associated to local CC mitigation actions

Figure 1.3 – Schematic representation of the different tasks to be developed in order to attain the thesis goal

The identification and categorization of the factors of change that may influence local GHG emissions (T1) can be seen as an analytical framework to build the evaluation methodology upon. This exercise is preceded by elementary causality tests between local actions and the respective energy system character-istics, in order to confirm the relevance of both higher-level policies and local specificities on observed changes.

In what concerns the quantification of the effects associated with local climate change mitigation actions (T2), this requires the disaggregation of the observed changes into effects associated to the different factors of change, including external factors and local climate change mitigation actions. This quantification model is then converted into a tool, in order to facilitate its application.

The methodology is then applied to case-studies (T3). This task is important to validate the previous exercises and to make small adjustments regarding the input data. This finetuning may include adjusting the level of detail of the data used, e.g. Moreover, the adaptation of input data to concur with data that is commonly available for the local level is an important step to ensure a wider applicability of the model by policy makers and other relevant practitioners.

(25)

1.6 Thesis structure 7

1.6 Thesis structure

Chapter 2 presents a review of methodologies for the evaluation of climate related policies, with a focus on local level. This includes the development of an analytical framework, with five characterizing features – moment of evaluation; geographical scope; object of evaluation; causal links consideration; and merit assessment –, which is based on the theoretical principles of policy evaluation. Existing approaches are then assessed according to these five characterizing features, to determine their adequacy to evaluate local climate change mitigation actions.

Chapter 3 is dedicated to an empirical study on the relation between local GHG emissions and local climate change mitigation actions. The influence of local actions on energy use and GHG emissions is tested using elementary causality tests and data from three distinct countries – Portugal, Sweden and United Kingdom. This chapter provides an insight on the relationship between the existence of local actions and observed change in the local energy system, as well as on the significance of the effects of external factors (as local specificities and national policy context) on local GHG emissions.

Succeeding the empirical insights on the relevant factors that influence local energy systems, chapter 4 provides a systematic review of the factors of change of local GHG emissions. This includes the identification of the relevant factors of change and their characterization with quantifiable indicators. This chapter provides a robust analytical framework for the development of the quantification model, being an intermediary step to the quantification of the individual effects of the different factors of change.

Chapter 5 is dedicated to the description of the model developed to quantify the effects of local climate change mitigation actions. It starts with a short discussion on the pros and cons of different methodolog-ical options, followed by the presentation of the chosen methodology to build the quantification model. The model itself is described in three steps: first, an overall description of the model and its operational features; second, a detailed description of the estimation of the individual effects per factors of change; and finally, a brief discussion on finetuning and validation of the results.

Chapter 6 is dedicated to the application of the model to Porto (Portugal), having 2004 and 2015 as base and reporting years, respectively. The case-study is described in detail, including the disaggregation of the observed changes into the effects per factor of change. Within this chapter, it is also presented the application of the model to six surrounding municipalities, in a less detailed manner, and the results are used as a tool for finetuning of the input data. The results obtained for the municipality of Porto are compared with the ones obtained for the other municipalities.

Chapter 7 provides a more detailed analysis of the results obtained for the municipality of Porto regarding the effects of local climate change mitigation actions. A cross-check between the actions implemented at local level (as reported by the local authority) and the results from the quantification model is performed, as a reality-check of the obtained results. It is also presented an analysis of the estimated effects in comparison with the projected outcomes, as established by the local authority. This can contribute for an assessment of the overall path towards climate change mitigation and the specific contribution of local actions.

Chapters 8 and 9 are the equivalent of chapter 6 and 7, respectively, for the municipality of Malmö (Sweden). The main objective of replicating the model for a second case-study corresponds to the

(26)

possibility of considering a larger timeframe (1990-2015) and the opportunity to obtain an additional validation of the model using a different context. In chapter 8, as for the first case-study, the model is also applied to six other municipalities.

Finally, the main conclusions are presented in chapter 10, followed by some reflections regarding the insights from the previous chapters on additional issues that could be subject for future research.

(27)

CHAPTER 2

Literature review

The main objective of this chapter is to perform a literature review on existing policy evaluation methods, assessing their adequacy to evaluate local climate change mitigation actions. Thus, it is intended with this review to understand if existing evaluation methods are adequate to assess local climate change mitigation actions and, if not, to identify what are the methodological characteristics that would guarantee the adequate assessment of these actions in terms of impact, efficiency and effectiveness. This is achieved by a review of the existing work on the evaluation of climate and energy policies taking into account the difficulties that come along with the assessment of the effects of local actions. In order to do so, an analytical framework was developed – considering not only the specificities of local actions’ evaluation but also general principles from evaluation theory.*

Section 2.1 presents the assessment framework upon which the review is structured. Section 2.2 introduces the studies that were assessed in this work and characterizes them according to the developed framework. Section 2.3 discusses this analysis according to their adequacy to evaluate local climate change mitigation actions. Section 2.4 provides an overview of the main limitations encountered in the reviewed studies, leading to concluding remarks and research directions for the work.

2.1 Analytical framework

In order to analyze existing approaches on their adequacy to assess local climate change mitigation actions, a framework that captures the critical features of evaluation was developed. According to Vaz et al. (2001), there are three types of questions that must be posed within the evaluation of environmental policies: descriptive; causal and normative.

- The descriptive refers to questions such as “What has happened? Or what will happen?” The way this question is answered depends not only on the time and the object of the evaluation approach, but also on

*The work presented in this chapter was already published in a peer-reviewed journal (Azevedo & Leal, 2017).

(28)

the geographical scope of the assessment.

- The causal refers to the links between the observed changes and the assessed policy. Different approaches may consider different causal linkages, resulting on a different decoupling of the effects.

- The normative refers to inquiries as “Are the results satisfactory?” Different approaches may answer differently to this question, or even not answer it at all. The existence (or not) of a merit assessment and how it is performed can characterize the different approaches on this matter.

Thus, within this chapter, and in order to consider the different aspects of evaluation, the analytical framework that was developed is composed by the following features: moment of evaluation; geographical scope; object; causal links consideration; and merit assessment. In this section, each feature is discussed individually, including the available options within the evaluation of local climate change mitigation.

2.1.1 Moment of evaluation

The time of each approach refers to the moment when the evaluation is performed in relation to the phase of the action’s implementation process, typically being ex-ante or ex-post.

An ex-ante evaluation corresponds to the assessment of a certain action or policy prior to its implementa-tion, i.e. a forward-looking assessment of the likely future effects. This type of approach is commonly used throughout the energy planning process, as a support tool for decision makers. It can be done through the use of models and the development of different policy scenarios, or, in simpler approaches, through the use of reference values from literature.

On the other hand, an ex-post evaluation corresponds to the assessment of the actual effects of a certain action or policy, subsequently to its implementation. Here, the evaluation focus on what was done and on the assessment of the respective consequences. “Ex-post evaluations can provide feedback information for policy-makers about the actual impact of a measure or policy and therefore are crucial for policy-learning. Results can be used to construct and calibrate evidence-based models and to underpin ex-ante evaluation.” (Vaz et al., 2001)

2.1.2 Geographical scope

The geographical scope corresponds to the boundary assumed by each approach, i.e. the physical area that is treated within the evaluation exercise. Despite being related with, this may not correspond exactly to the level of the policy actions assessed. For instance, there might be some studies that intend to evaluate the impact of European Union policies in a specific national territory or even in a given municipality. Nonetheless, when different from the administrative level of the policy assessed, the geographical scope is usually smaller than the latter, as the effects of a policy at higher governance levels (than the level of implementation) are usually not that significant.

(29)

2.1 Analytical framework 11

2.1.3 Object of evaluation

Evaluation approaches can differ significantly in terms of their focus (i.e. evaluation object), in the sense that they can either concentrate on the operational issues or on the effects of the action(s) assessed. The first refers to studies that focus on the policy mechanisms as the object of research – looking at features as innovation, stakeholders’ involvement, etc. This type of analysis is mostly performed when the implementation is complete, in order to identify the most relevant advantages and shortcomings and how the mechanism itself could be improved.

In contrast, the studies that intend to evaluate policy outcomes aim at quantifying the effects of the policy assessed, either ex-ante or ex-post. Here, the central focus is the identification and evaluation of the effects that derive from the action(s) being assessed. Within this type of studies, there are different approaches in terms of scope, i.e. which types of effects are considered. Policy effects may be divided in three major categories: output effects, outcome effects and impact effects. The first refer to the tangible results of a measure; the second refer to the response of the target group to the output; and the last to the overall influence of the measure in the policy issue.

2.1.4 Causal links consideration

The evaluation of what happened (or will happen) following the implementation of a certain policy action does not necessarily correspond to the assessment of the actual effects of that policy (Gysen, 2006). Due to the universal scope of climate policy, its effects are often coupled with the effects of other non-related policies, or even to related policies originated from different governance levels. For instance, the shift towards tourism as the main economic activity, although an economically driven decision, can have a negative impact regarding climate change mitigation as it is an energy intensive activity. Thus, the observed changes do not correspond to the effects of a specific policy but instead to a bundle of effects from different policies and/or autonomous trends. The consideration of causal links refers to the process of decoupling the observed changes into the effects of different policies or actions.

In the case of local climate change mitigation actions, there are several external factors that can be considered, from past and ongoing policies at different governance levels to the evolution of local context occurring simultaneously to the changes caused by local actions. Indeed, the issue of policy interactions is seen as especially relevant for local climate policies, as the effects of their own climate change mitigation actions depend not solely on other local policies but also on the existing higher-level policies. For example, national fuel taxes can either favor or hinder local policies that promote the use of public transportation in detriment of private car. Moreover, recent studies showed that the local context can enable or constrain local authorities’ actions, and that the verified outcomes can be altered by exogenous changes to the local conditions (Ostrom, 2005). There is also empirical evidence on the importance of such factors in the success (or failure) of local policies – whether or not climate-related.

(30)

2.1.5 Merit assessment

Evaluation of a policy or action differs from its quantification of impacts in the sense that evaluation implies the comparison of the assessed impacts with implicit or explicit standards (Weiss, 1998), some-what implying a benchmarking. Within this framework, this comparison and respective interpretation is denoted as merit assessment. Different evaluation approaches can present very different merit assessments, depending on the purpose of the study and the type of policy assessed. For instance, policy actions can be assessed in terms of their effectiveness, relevance and/or efficiency.

2.2 Evaluation approaches addressing climate change mitigation actions

As the scientific community and the overall community have acknowledged the potential contribution of local actions for climate change mitigation, various studies were performed on the topic – studying the relationship between municipalities and climate change (Broto & Bulkeley, 2013). Here, the focus is on the studies that may contribute to the assessment of local climate change mitigation actions, even if this is not their main goal. Moreover, considering that certain approaches that have been used at different policy levels may also be applied to assess local climate change mitigation actions, the review was enlarged to cover the evaluation of higher-level policies.

The selection of studies was performed through a systematic search using the following keywords: local level policy evaluation; local energy planning; climate/energy policy evaluation; climate/energy policy assessment; and climate/energy policy interactions.

The reviewed literature provides a range of diverse methodologies – case-study analysis, statistical analysis, survey research, model building, etc. – in an effort to incorporate different features of climate and energy policy instruments bound to affect their impacts. An overview of the studies reviewed in this work is presented in Table 2.1, structured according to main objective.

(31)

2.2 Evaluation approaches addressing climate change mitigation actions 13

Table 2.1 – Studies included in the review, according to their main goal

Main Goal Studies / Methodologies

(1) Energy Solutions for Smart Cities and Communities . DG Energy, 2014.

(2) Evaluating the effectiveness of urban energy conservation and GHG mitigation measures: The case of Xiamen city . Lin et al., 2010.

(3) Do city climate plans reduce emissions? Millard-Ball, 2012.

(4) Analyzing the effects of Energy Action Plans on electricity consumption in Covenant of Mayors signatory municipalities in Andalusia . Pablo-Romero et al., 2016

(5) Evaluating public policy mechanisms for climate change mitigation in Brazilian building sector . Augustus de Melo et al., 2013.

(6) Local authorities in the context of energy and climate policy . Comodi et al., 2012.

(7) How to develop a Sustainable Energy Action Plan . Guidebook Part 3 . Covenant of Mayors, 2010.

(8) A multi-criteria evaluation method for climate change mitigation instruments . Konidari and Mavrakis, 2007.

(9) A comparative analysis of urban energy governance in four European cities . Morlet and Keirstead, 2013.

(10) Decision support methodology for local sustainable energy planning . Neves, 2012. (11) A quantitative method for the evaluation of policies to enhance urban sustainability . Fitzgerald et al., 2012.

(12) An analytical method for the measurement of energy system sustainability in urban areas . Jovanovic et al., 2010.

(13) A survey of urban climate change experiments in 100 cities . Broto and Bulkeley, 2013. (14) Local Government and the Governing of Climate Change in Germany and the UK . Bulkeley and Kern, 2006.

(15) Local climate action plans in climate change mitigation - examining the case of Denmark . Damsø et al., 2016

(16) Analysis of the Covenant of Mayors Initiative in Sicily . Famoso et al., 2015.

(17) Local governance and climate change: Reflections of the Swedish experience . Granberg and Elander, 2007.

(18) Moving from agenda to action: evaluating local climate change action plans . Tang et al., 2010. (19) State and Municipal Climate Change Plans: The First Generation . Wheeler, 2008.

(20) Assessing sector-wide programs with Statistical Impact Evaluation: A Methodological Proposal . Elbers et al., 2008.

(21) Coherent or inconsistent? Assessing energy security and climate policy interaction within the EU . Strambo et al., 2015.

Evaluation of effects

Impact assessment in the context of energy planning Sustainability assessment Policy mechanisms assessment Policy interactions

(32)

There are few studies specifically focusing on the assessment of local climate change mitigation actions. The majority of these correspond to the individual assessment of case studies – mostly individual actions or small sets (Lin et al., 2010, EU, 2014). Here, the consideration of interactions and the decoupling of the actual effects that are due to the assessed policy are quite limited. The studies that present a systematic tool to identify the accountability of local actions usually do not consider the potential impact of local externalities nor the interaction with higher-level policies (see, e.g. Millard-Ball (2012a)). More recently, appeared the first studies that attempt to analyze the effects of local actions, (partially) considering the existence of external factors (Pablo-Romero et al., 2016). The full assessment of these studies according to the evaluation framework is presented in Table 2.2.

Table 2.2 – Analysis of the evaluation approaches used in studies focusing on evaluation of effects (numbering of the studies corresponds to the numbering of Table 2.1)

Moment Scope Object

Evaluation

Method Causal Links Considered

Merit Assessment (1) Ex-post Local level Effects / Policy

mechanisms

Qualitative and

Quantitative None No

(2) Ex-ante Local level Effects Quantitative (LEAP model)

- The impact of economic, social and technological evolution on projected policy effects

No

(3) Ex-post Local level Effects Quantitative - The impact of observed changes on

the impact of climate plans No

(4) Ex-post Local level Effects Quantitative

- The impact of individual income, population density and size of the municipality and existence of a SEAP, on local electricity consumption

No

Local energy planning has been emerging as a relevant research topic, possibly prompted by the initiatives – as the Covenant of Mayors and Cities for Climate Protection – that promote the development of local energy action plans and the high adherence by local authorities. In parallel with the development of methodologies to create local energy plans, there are also several studies on the estimation of potential effects of different actions (Comodi et al., 2012, CoM, 2010a, Morlet & Keirstead, 2013, Neves et al., 2015). As the main goal of these studies is to support decision makers in local energy planning, the assessment approaches are usually ex-ante, i.e. they provide an estimate of what could be the effects of different actions. Even though most of these studies consider the natural evolution through the development of a business-as-usual scenario – taking into account projected socio-economic changes and the evolution due to ongoing energy and climate policies – the consideration of causal links is often limited. This might be due to the high level of uncertainty associated with the projections, which makes it difficult to predict the potential interactions in detail. Another common drawback of these studies is the assessment of the various actions individually, i.e. not accounting for the potential interaction between the different actions. Indeed, the effect associated with the implementation of stricter building regulations combined with incentive-based programs to improve the energy efficiency of space heating equipment should be smaller than the sum of the effects associated with the implementation of these actions separately. However, this type of interactions is frequently disregarded when estimating the effects of a set of distinct actions.

(33)

2.2 Evaluation approaches addressing climate change mitigation actions 15

In parallel, at national level, there are also several studies that focus on the assessment of national climate change mitigation instruments with the intent of assisting national governments in policy making. In general, their features are very similar to what is said above regarding local level energy planning methods. Konidari & Mavrakis (2007) is different in the sense that the assessment is performed ex-post and so the quantification of effects is based on empirical data. On the other hand, Augustus de Melo et al. (2013) presents an ex-ante assessment that is worth mentioning due to the consideration of the potential interactions between the assessed policy and the existing national policies.

The full assessment of these studies according to the evaluation framework is presented bellow in Table 2.3.

Table 2.3 – Analysis of the evaluation approaches used in studies focusing on impact assessment in the context of energy planning (numbering of the studies corresponds to the numbering of Table 2.1)

Evaluation

Merit Assessment (5) Ex-ante National level Effects

Quantitative (MACC) Qualitative (MAC)

- The impact of existing national

policies on projected policy effects No

(6) Ex-ante Local level Effects Quantitative

No

(7) Ex-ante Local level Effects Quantitative None No

(8) Ex-post National level Effects Quantitative

(MCA) None Yes

- The impact of existing demand for energy across time and place on projected policy effects

No

Ranking of different alternatives

Concerning the studies focusing on the assessment of sustainability at local level, these aim at both oper-ationalizing the concept of local sustainability and developing methodologies to allow its quantification– including the definition of indicators and respective calculation methods (see Neves & Leal (2010) for a review). Some of these studies attempt to measure the sustainability increase due to local actions (Fitzgerald et al., 2012, Jovanovic et al., 2010). While this exercise could be very valuable in assessing the impact of local actions, its relevance is reduced as the assessed increase corresponds to the difference between the situation before and after the local action, without considering any causal links between the observed changes and the assessed policy actions. The full assessment of these studies according to the evaluation framework is presented next (Table 2.4).

There are also several studies focusing on the competences of local authorities and their potential for action (Bulkeley & Kern, 2006, Granberg & Elander, 2007, Wheeler, 2008, Famoso et al., 2015, Damsø et al., 2016), as well as on the challenges they face in terms of policy implementation and effectiveness (Gysen, 2006, Tang et al., 2010) – based on both theoretical grounds and empirical data. In these studies,

(34)

Table 2.4 – Analysis of the evaluation approaches used in studies focusing on sustainability assessment (numbering of the studies corresponds to the numbering of Table 2.1)

Evaluation

Merit Assessment (11) Ex-ante Local level Effects Quantitative - The impact of local characteristics on

sustainability enhancement

Ranking of a set of policies

- The impact of economic, social and technological factors on future energy needs

Ranking of different scenarios

the policy mechanisms are the actual object of research, focusing on features as innovation, stakeholders’ involvement, and their dependence from higher-level governance structures. In general, given the nature of these studies, the policy actions are assessed qualitatively, and no merit assessment is performed (i.e. there is no comparison of the observed outputs with any implicit or explicit standards). Tang et al. (2010) is an exception in the sense that it provides a quantitative analysis of the “quality of different local action plans”, through a statistical analysis of different performance indicators. Even though, the analysis is focused on the action inputs (i.e. what is included in the plan) and not on the actual effects resulting from the plans’ implementation. Thus, even if performed ex-post, the approaches presented by this type of studies do not assess the effects of local policy actions, and, consequently, nor the contribution of these actions towards climate change mitigation. The full assessment of these studies according to the evaluation framework is presented in Table 2.5.

Table 2.5 – Analysis of the evaluation approaches used in studies focusing on policy mechanisms assessment (numbering of the studies corresponds to the numbering of Table 2.1)

Evaluation

Merit Assessment

- The impact of The targeted sector on The form of innovation

- The impact of the leading actor on where, when and what is done

(14) Ex-post Local level Innovation / Policy

mechanisms Qualitative None No

(15) Ex-post Local level Policy mechanisms

/ Plans Qualitative None No

(16) Ex-post Local level Policy mechanisms Quantitative and Qualitative

- The impact of local context and higher-level framework on the degree of commitment from local authorities

No

(17) Ex-post Local level Policy mechanisms Qualitative None No

(18) Ex-post Local level Policy mechanism

/ Plans Quantitative None Yes

(19) Ex-post Local level Policy mechanisms

/ Plans Qualitative None No No

(13) Ex-post Local level Innovation /

Origin of actions Qualitative

Finally, the assessment of different climate and energy policy instruments in a more realistic policy interaction environment has been prompted by the increased policy-makers awareness of the issue

(35)

2.3 Discussion on the adequacy to evaluate local climate change mitigation actions 17

of overlapped policies (Boonekamp, 2006). Within the studies focusing on policy interaction, both quantitative and qualitative evaluation approaches are used. While the latter mainly contribute to the assessment of the diversity and complexity of policy interactions affecting the impacts of the policy instruments being assessed, the quantitative approaches provide absolute numbers and economic trends that affect them. Strambo et al. (2015), being representative of the existing qualitative studies, presents some limitations, namely: the use of static representations – disregarding the evolution of socio-economic and technological context over time, and the favoring of ex-ante evaluation over ex-post – while the latter can reflect reality to a higher extent. Despite not being energy and climate related, Elbers et al. (2009) is a good example of an assessment of policy actions considering the causal linkages between observed changes and policy effects; however, no merit assessment is performed. The full assessment of these studies according to the evaluation framework is presented in Table 2.6.

Table 2.6 – Analysis of the evaluation approaches used in studies focusing on policy interactions (numbering of the studies corresponds to the numbering of Table 2.1)

Evaluation

Merit Assessment

(20) Ex-post National level Effects

Quantitative (Statistical Impact

Evaluation)

- The impact of other policies interventions and natural evolution (+ not considered in the analysis) on observed changes

No

(21) Ex-ante EU level Objectives /

Operation Qualitative

- Between Energy Security policies and

CC Mitigation policies at EU level No

A more detailed discussion on the individual features and its implications for the evaluation of local climate change mitigation is presented in the following section.

2.3 Discussion on the adequacy to evaluate local climate change mitigation

actions

The review of existing studies that contribute to the assessment of local climate change mitigation actions showed very different approaches that have different implications in terms of the subsequent evaluation. These differences can be a starting point to the discussion on the adequacy of different approaches to assess the effects/contribution of local actions towards climate change mitigation. In this section, the implications of the available options are discussed for each characterizing feature individually.

2.3.1 Moment of evaluation

From the review presented in the previous section, it seems that both ex-ante and ex-post approaches are widely used for the assessment of local climate change mitigation actions. Nonetheless, it is also noticeable that the choice regarding the timing is closely related with the studies’ goals – while ex-ante approaches are commonly used for energy planning related studies and for the assessment of policy interactions,

(36)

the ex-post approaches are mostly adopted in the evaluation of specific case-studies and in sustainability assessments (with causal links consideration).

Not disregarding the importance of ex-ante approaches in supporting decision makers within energy and climate related planning, the uncertainty associated to these methods constitutes a barrier to the assessment of the actual effects of any policy action (Vaz et al., 2001). The contribution of local actions to global climate change mitigation may be intensely influenced by innumerous factors (being or not local specific) which makes the estimation of effects prior to implementation even more difficult. Instead, ex-post evaluations can use information regarding what was done and observed changes to minimize the uncertainty associated to the assessment of the actual effects of local actions.

Moreover, considering that there are several actions whose implementation process is ongoing or already finished, there is a continuously increasing amount of data available on the actions local authorities are undertaking. This empirical data can be extremely valuable to understand and isolate the actual effects of local actions from other changes in the local energy system. Both top-down and bottom-up evaluation methods may use such empirical data as an input to estimate gross-to-net correction factors in order to derive the net effect of assessed policy actions from observed changes. The choice of method highly depends on the available data (EMEEES, 2009).

The lack of harmonization of existing empirical data could be a concern, especially when using top-down evaluation methods and comparing the outcome of different policy actions, as in Pablo-Romero et al. (2016) and Tang et al. (2010).

2.3.2 Geographical scope

With this review, it became clear that there are already several studies that focus on policy actions implemented at local level, promoted or not by local authorities. However, the assessment of the associated effects is not that common, mostly done through case-study analysis and the assessment of individual actions (within the context of energy planning, e.g.). Indeed, compared to what has been done on the quantification and evaluation of the results achieved by national and international policy actions, the existing work regarding the evaluation of local actions effects is still scarce.

Within the assessment of local policy actions, the geographical scope doesn’t need to go beyond the administrative boundaries of the local authority very often, as its effects are usually not significant outside this area. There might be some exceptions, as transport related actions e.g., where policy actions may affect the whole metropolitan area; but in these cases, the local authorities of the surrounding municipalities are usually also involved in the decision-making process.

2.3.3 Object of evaluation

Similarly to what is found in general policy evaluation studies, there are two distinct perspectives in which to consider the evaluation of (local) climate change mitigation actions – either focusing on the operational issues or concentrating on the effects resulting from the policy implementation. The methods focusing on the assessment of the policy mechanisms’ operational issues are usually qualitative and based on surveys

(37)

2.3 Discussion on the adequacy to evaluate local climate change mitigation actions 19

and/or interviews. For instance, Broto & Bulkeley (2013) presents evidence from the analysis of a survey of several urban climate change experiments, focusing on type of instruments and innovation. Contrarily, the studies focusing on the assessment of effects are mostly quantitative, independently of being based on theoretical principles (as in Neves et al. (2015)) or empirical data (as in Pablo-Romero et al. (2016)). Both evaluation perspectives are extremely important, contributing differently for the improvement and adaptation of future or ongoing policies. On one hand, the studies focusing on the policy mechanisms and their implementation usually provide insights regarding the barriers found along the implementation process, and factors that may contribute (or not) to their success. This may be useful to understand which conditions could facilitate or hinder the implementation of different policy mechanisms, providing a framework to match local conditions with the most adequate policy mechanisms. On the other hand, the evaluation of policy actions’ effects is crucial to assess their contribution towards climate change mitigation. This quantification is relevant not only for the individual assessment of local actions, but also for the comparison of different individual actions and of different municipalities’ overall efforts. Within the approaches that focus on the evaluation of policy effects, some variations regarding the effects considered were also found – outputs, outcomes and/or impacts. While they are all useful in the assessment of policy actions, only the quantification of the impacts fully reveals the contribution of the actions towards climate change mitigation, as this corresponds to the overall influence of the action in the policy issue.

2.3.4 Causal links consideration

The importance of identifying the causal links between observed (or projected) changes and the policy actions being assessed is recognized by innumerous studies. Within these studies several factors are referred to as important for the evaluation of local actions, namely: (1) the local context, including legal competences and financial and human resources of local authorities; (2) the socio-economic changes observed at local level; and (3) exogenous factors such as autonomous savings and other policies (at local and higher administrative levels). However, there are only a few that include them in the quantification/assessment of the policy effects.

Local context refers not only to determinant characteristics of local authorities, as legal competences and financial capacity, but also to certain attributes of local inhabitants, as awareness towards climate change and willingness to act, that may influence the outcome of different policy actions. These factors are mainly considered in the studies that focus on the assessment of different policy mechanisms, as in Bulkeley & Kern (2006). Here, there is an attempt of understanding the link between local context and innovation of policy instruments.

The evolution of the (local) characteristics, prompted by economic, social and technological changes, is the external factor that is most commonly considered in the reviewed studies. In ex-ante approaches this is usually done through the development of a business-as-usual scenario, while ex-post approaches seem to prefer the use of empirical data to build indicators that reflect these changes. Both approaches consist on understanding what would happen (or would have happened) if local actions would not take place; because, even in the absence of policy actions, the local energy system is continuously evolving with time.