Theory of Change

Instead of scoring the programme impact, the evaluation provides a discussion on the intervention logic that underpinned the implementation of WSPST. The aim is to explore what WSPST was trying to achieve during its three phases, and how the design of the Programme supported the achievement of these objectives.

WSPST used Logical Frameworks to describe the intervention logic, i.e. to explain how the Overall Objective (or Impact in results-based management terminology) is to be achieved. Each phase had its own logical framework. Based on the information provided in the Programme Documents, we have provided a summary of each level in the results “hierarchy” outlined in Appendix 3.

The TOC (Appendix 2) is based on the original logical frameworks for the three phases of WSPST (see Appendix 3). In addition, a number of components that had been implicitly part of the intervention logic but that had not been reflected in the formal planning documents were included in the analysis of the intervention logic. This approach allowed the team to visualise and discuss the “why” and “how” the programme was, or should have been, implemented. It also formed the backbone of the evaluation in terms of organising the evaluation questions in a logical sequence and linking them into the evaluation criteria used in the assignment.

Objectives of WSPST

In Phase I, the Overall Objective (Impact) was centred around poverty reduction, measured by percentage of poor households and their average income. The goal would be achieved by the provision of potable water and hygienic environment for the benefit of urban populations and economic activities (programme Purpose/Outcome). Elimination of slums and temporary housing in all cities and towns is also mentioned; however, the means or ways by which that objective would be achieved is not described in the programme document (MFA, 2003b).

In Phase II, the focus was no longer framed explicitly around poverty reduction. The Overall Objective (Impact) and Purpose (Outcome) were brought to a similar level; access to safe water and wastewater treatment measured by the percentage of urban and rural populations as well as industrial zones (indicators of the Overall Objective).

The Overall Objective (Impact) of Phase III adds the element of contribution to the social and economic growth in WSPST towns in addition to the “mere” improved WS and DS systems. The indicator sounds ambitious at first (100%), but in fact, it “only” refers to 100% of the service area of the schemes, not the whole population living in the town. Also, the logical framework does not include an indicator for “social and economic growth” (MFA, 2013b).

The Purpose (Outcome) of Phases I and II are similar, with the emphasis on construction of WS and DS schemes. This level of the logframe was changed in Phase III when the focus became

“improved awareness of the final beneficiaries regarding their rights to water and sanitation (populations only) and regarding the risks of untreated water and wastewater and of unhygienic

environment (populations and enterprises)” to reflect the HRBA approach that had received increased attention in the MFA during the same period. In addition, Phase II includes the

“Integration of Finnish Revolving Water Fund into National Revolving Water Fund” (MFA, 2009).

The analysis shows that WSPST is broadly in line with poverty reduction objectives. However, a closer investigation to the Programme logic reveals a lack of clear vision on what the Programme is supposed to achieve and how success would be measured. Is the objective to improve poor people’s health (e.g. reduce water borne diseases)? Is it to increase their income? Is it to improve environmental standards (e.g. pollution levels in the nearby river)? If only those people who live in the WS/DS scheme service area are covered, what should happen with those residents that live within the town but do not have access to a service? Are enterprises part of the beneficiaries and what type and size of enterprises? In other words, objectives were not clearly prioritised, the indicators and sources of data were vague and/or absent. In addition, the Programme did not possess an M&E system, which would have provided an opportunity to provoke reflections around the issue. No baseline data was collected by WSPST that could support impact measurements.

WSPST Theory of Change

The evaluation team has examined the underlying logic that steered the Programme from its start by constructing a Theory of Change (TOC) based on the programme documents and interviews with MFA staff (see Appendix 2).

In this process, the team added relevant elements that were not part of the original logical framework. For example, it is not emphasised in the Programme documents that the whole model of WSPST is based on the idea that the Finnish Water Revolving Fund would allow replicating and scaling up the construction of WS and DS schemes in other small towns in Vietnam. However, that is what a revolving fund is essentially set up to do. Therefore, we have sought a balance between those different explicit and inexplicit elements of the intervention with the purpose of trying to make a useful TOC for dialogue during and after the evaluation.

Starting from the left-hand side, the TOC sets the stage by listing the main inputs in WSPST. In addition to the financial and technical assistance, the TOC includes also political leverage, which is one of the key elements of Finland’s Country Strategy Modality under which bilateral programmes are implemented. Political support from GOV has also been added: it is assumed that the role of MOC at central level is critical in motivating provinces to step up their action for achieving high coverage of WS/DS services.

The following column (Components) links the TOC to the main intervention areas of WSPST, namely: WS/DS Schemes and support to strengthening the enabling environment in Vietnam.

The Results (Outputs) column - which corresponds to the evaluation criterion of Efficiency - describes the tangible outputs that the Programme is expected to deliver and to which it has a high degree of influence. These include physical infrastructure (including the laboratory), training stakeholders to operate and maintain the facilities, and the social marketing efforts. At this level, the Programme aimed also at supporting the passing of new laws related to WS and DS.

Overall, the observation is that the overarching Programme logic approached the needs of the populations of small towns in Vietnam from the perspective of strengthening physical infrastructure instead of adopting primarily a capacity development approach although the latter constituted an important element of the intervention through the provision of international TA to Vietnamese stakeholders. For this reason, we have highlighted the Programme’s “core area”

with a dotted line in the TOC. We have also located the boxes on training courses at the same level with the building of the infrastructure projects because WSPST carried out both in parallel.

At the level of Outputs, it is expected in the TOC that all infrastructure has been built, facilities are functioning, all the relevant people have been trained and informed, residents of the towns

have been given the means to connect to the schemes, and everything is in place so that water and wastewater can continuously flow in the schemes. Also, gaps in national legislation have been filled by removing barriers for successful implementation of the Programme and the sector as a whole. Finally, the TOC is based on the assumption that FRWF has been integrated with NRWF to ensure sustainability of the financing system.

As will be discussed later in the report, the fact that that the DS schemes did not meet quality requirements and/or could not be maintained, caused the logical chain to become broken at the level of outputs for DS schemes. WS scheme implementation achieved satisfactory quality, as is explained in its corresponding section of the evaluation report.

The programme design includes a number of assumptions which, as discussed in the following paragraphs, relate to both management, financial as well as social aspects of the interventions.

A key element at the output level was the enabling environment, which included the assumption that either sufficient laws are in place in Vietnam or that WSPST will be able to address the gaps early enough in order not to jeopardise the implementation of the Programme. Given that the main approach of the Programme was to focus on infrastructure and to provide technical assistance alongside the procurement and construction processes, the institutional aspects remained rather an additional element to the intervention than its main entry point. In that sense, the Programme relied on the expectation that the WSPST governance system and careful attention to each step of the procurement of the works would suffice to ensure quality and sustainability.

One example is the fact that the DS schemes’ O&M financing relied on TPC/PPC commitments rather than national legislation that would oblige households to connect to the DS schemes and to pay a wastewater fee that would be sufficient to ensure financial sustainability of the schemes.

For further discussion on this aspect, refer to Section 3.5 on Sustainability.

The initial vision was that a NWRF would be established with the leadership of the GoV. As the context in which WSPST pursued its activities changed, the assumption did not materialise as the NWRF was never established. Other donors that were initially interested in investing funds in the NRWF together with Finland pulled out leaving debilitating the momentum towards the establishment of the Fund. This aspect was unexpected, and it affected negatively the long-term ambition level of WSPST. Furthermore, the GOV equitization policy for WSCs has made the initial concept of the NRWF redundant. Additional analysis on FWRF is provided in Sections 3.2 to 3.5.

Appendix 16 provides a timeline for the FWRF.

At the Purpose (Outcome) level – which corresponds to the OECD evaluation criterion of Effectiveness - the picture becomes more complicated as the Programme’s influence is gradually reduced. At this level, the Programmes influence on the results becomes gradually weaker compared to Output level meaning that the actors need to feel motivated to support the changes, and the conditions need to be favourable for the changes to happen. For the purpose of this evaluation, we have distinguished between immediate and long-term outcomes to better allow unpacking of the intervention logic.

In the first Outcome column of the TOC, we describe a system where the WS companies and DS operators, as well as local government, all fulfil their own roles (institutionally, socially, technically and financially) to ensure WS and DS schemes are operated and maintained properly and constantly over long periods of time. In addition, information flows between the customers and the operators and grievances are addressed.

The assumptions at this stage include that citizens connect to the WS and DS schemes, pay for and use the services. For DS, the logical chain was broken already in an earlier stage (poor designs), which is one of the reasons for the low connection rates. The long-term outcome for the FWRF is that it allows replication of the model in new towns. However, the small size of FWRF

allows only one town to be covered per year, leaving its importance relatively small in the scale of Vietnam.

When it comes to the Impact-level, the main challenges are that, in addition to not defining clearly what the expected impact should be, the WSPST did not collect baseline data, or put in place an M&E system. If the expected impact would have been improved health of target populations, information on the health status of the inhabitants benefiting from WS and/or DS schemes should have been collected prior to the intervention. If it was increased economic activity, there should have been estimates on the number of businesses in the towns, or other relevant information. In the case of wastewater treatment, e.g. measurements of pollution levels should have been gathered. However, the evaluation team’s understanding is that there was no collection of such baseline data.