(Article title page)
Relationships between NPM, the use of PM practices and the organizational performance:
empirical evidence from government agencies
Patrícia S. Gomes
PhD in Accounting – The University of Minho
Assistant Professor at the Polytechnic Institute of Cávado and Ave (E-‐mail [email protected])*
Sílvia Camões
PhD in Political Science – State University of New York -‐ Binghamton Associate Professor at the University of Minho
(E-‐mail [email protected])
João B. Carvalho
PhD in Management Sciences – The University of Zaragoza Associate Professor at the University of Minho
(E-‐mail [email protected])
*Corresponding author
Key words: NPM reforms; performance management practices; organizational performance;
government agencies; survey method.
Article classification: research paper.
Abstract:
• Purpose – this paper has two main purposes: (1) explore if government agencies more oriented to NPM postulates are more willing to use PM practices and to improve their performance; and (2) investigate whether the fit between the use of PM practices and the organizational performance is dependent upon from the capacity of agencies to adapt its structures to changes introduced by NPM reforms.
• Design/methodology/approach -‐ this paper is based on the survey method and provides empirical evidence from Portuguese government agencies.
• Findings (mandatory) -‐ Our findings suggest that government agencies that made structural arrangements under the NPM reforms are more willing to use PM practices and will perform better than other agencies. In addition, this paper show that the
relationship between the use of PM practices and the organizational performance is dependent upon from the agencies capacity to adopt new structural arrangements under the NPM reforms.
• Practical implications – this paper has three main contributions: (1) contribute to knowledge about the relationship between the introduction of NPM changes in the use of PM practices; (2) contribute to clarify whether agencies more oriented to NPM postulates are improving performance; and (3) help to clarify the way the organizations should adapt their structures for to be more effective in the use of PM practices.
• Originality/value -‐ The quantitative empirical research, based on the unique survey applied to Portuguese government agencies on this field, allow us to add to prior research mainly based on case studies and oriented to local governments (Budding, 2004).
Relationships between NPM, the use of PM practices and the organizational performance:
empirical evidence from government agencies
1. Introduction
In the early 1980s a new wave of reforms in the broader public sector started across most OECD countries (Osborne and Gaebler, 1993; Pollitt and Bouckaert, 2004) which were central to the rise of the ‘New Public Management’ (Hood, 1995). The NPM paradigm proposed a radical new focus on service delivery and cost-‐effectiveness in order to change public sector organization and culture. ‘Accountability’ is one of several concepts that became central to the new public management and is a complex subject (Robinson, 2003).
From the agency standpoint, it makes sense to talk about a parent/subsidiary relationship in which the parent assign the responsibility and the subsidiary is responsible to the parent for performing the task (Robinson, 2003). The shift from legal accountability to managerial accountability introduced the focus on the concepts of efficiency and the effective use of resources (Sinclair, 1995; Robinson, 2003) as well as on performance management (Johnsen, 2005). In consequence, we assist to the processes of “agencification”, which imply that executives/managers became more directly and clearly accountable in terms of their ability to produce results and to run their organizations efficiently.
On the other hand, after the ‘agencification’ process the increase of management autonomy has been incentivized as a way to increase the effectiveness and efficiency of public entities and to make them more oriented to results and achievements (Osborne and Gaebler, 1993;
Hood, 1991 and 1995). Therefore, the basic postulate is that decentralization and power delegation to units/agencies should help increase the responsiveness, effectiveness and efficiency of the public sector (Eckardt, 2008). In addition, the more freedom for the units/agencies also meant an obligation for them to provide more and better information about results and to account for their actions (ter Bogt and van Helden, 2000; Budding, 2004).
Although ‘accountability’ and ‘being held accountable for results’ became key elements of NPM (Hood, 1995), little is known about their effects on the use of performance management (PM) practices and on the organizational performance. In fact, some doubts appear in literature about the NPM consequences (Brunsson and Olsen, 1993; Groot and
Budding, 2008; Lapsley 2008 and 2009). Brunsson and Olsen (1993) talk about unintended consequences of NPM reforms which, in their opinion, are attributed to the complexity of public service organizations. Lapsley (2009) also talk about disappointments on the NPM consequences. In its opinion, the NPM adoption is based ‘on governments having faith in its deployment to transform their public sectors using private sector performance criteria’
Lapsley (2009: 1). In addition, based on four criteria (the role of management consultants, the development of e-‐government, the emergence of the ‘audit society’ and the increasing importance of risk management), Lapsley (2009) states that the use of NPM is often a cruel disappointment for governments.
Although some problematic issues in held managers accountable in non-‐profit organizations, given the political-‐administrative system features, and some mixed results provided by prior studies about the success of the agencification process, we argue in this paper that agencies more oriented to NPM postulates are more willing to use PM practices and to improve their performance. Therefore, this paper aims to investigate whether the fit between the use of PM practices and the organizational performance is dependent upon from the capacity of agencies to adapt its structures to changes introduced by NPM reforms. Following these research objectives, we feature government agencies in two sub-‐samples according their higher/lesser compliance with NPM postulates: the NPM agencies and the non-‐NPM agencies (based on the methodology adopted by Budding, 2004).
This paper aims contribute in different ways. First, contribute to knowledge about the relationship between the introduction of NPM changes and the use of PM practices. The only known studies on this relation are of ter Bogt and van Helden (2000), Budding (2004), and Arnaboldi and Azzone (2010). However, these studies are not survey researches which difficult the generalization of findings. The quantitative empirical research, based on the unique survey applied to Portuguese government agencies on this field, allow us to add to prior research mainly based on case studies and oriented to local governments (Budding, 2004). On the other hand, prior studies show mixed results on the effects of NPM postulates on the organizational performance (Brunsson and Olsen, 1993; Groot and Budding, 2008;
Lapsley 2008 and 2009). This paper aims also contribute to clarify whether agencies more oriented to NPM postulates are improving performance. Third, a practical contribution of
the paper is help to clarify the way the organizations should adapt their structures for to be more effective in the use of PM practices.
The remainder of the paper is organized as follows. The next section presents the background of the literature review as well as the research questions explored in the paper.
The section three presents the research design, focusing the context and features of Portuguese government agencies, the employed research method and the measurement of NPM variable. In the section four and five we present descriptive statistics for different variables as well as the empirical results for to answer to each research question. The paper concludes with the discussion of the findings and the presentation of the main conclusions.
2. Background and research questions
The turbulent environment faced by most public organizations after NPM reforms has forced top managers to rethink their internal organizational structure necessary to improve performance (Abernethy and Lillis, 2001). The great attention has focused on the new structural arrangements needed to improve efficiency and effectiveness of public government actions. Therefore, we assist from the eighties to a shift from the traditional centralized organizational structure by a more decentralized and accountable structure in order to increase the autonomy in the policy execution at the level of divisions and units (agencies in this paper). In consequence, these new arrangements increase the obligation to produce more and better performance information (ter Bogt and Helden, 2000; Budding, 2004; Johnsen, 2005). As said by Wynen and Verhoest (2012), because the freedom to manage a greater emphasis has been put in results control. This way, more sophisticated PM practices are needed as a result of NPM reforms.
Performance management is considered as a main pillar of public management reforms in last years (Heinrich, 2002; Modell, 2005; Moynihan 2006; Walker et al., 2011) who emphasises management by objectives (per Laegreid, et al., 2006) and ‘managing for results’
(Modell, 2005). For Robinson (2003: 179), setting goals and measuring performance are elements of the scientific management tradition. In this paper, for PM practices we intend the setting and specification of goals to achieve, the measurement and evaluation of achievements, the allocating of decision rights and the rewarding of performance (see for example, Otley, 1999; Ittner and Larcker, 2001; Heinrich, 2002; Verbeeten, 2008).
The literature on government performance measurement has called the attention to some problematic issues in the implementation of PM practices that can lead to dysfunctional consequences (see for example, Pallot, 2001; De Bruijn, 2002; Johnsen, 2005; Modell, 2005 and 2009; Vakkuri and Meklin, 2006; ter Bogt, 2008 and Verbeeten, 2008). For example, the ambiguity of objectives, the reduced measurability of outputs, the repetitiveness of activities and the unknown process of transformation can make difficult to use output and outcomes measures to control organizations. As was stated by Modell (2005: 58), ‘outcomes indicators seem to have been those least used for controlling public sector organizations’, although they are ‘pivotal’ for assess its effectiveness. In addition, the ambiguity of objectives decreases the extent to which politicians and managers can be held accountable (Johnsen, 2005; Vakkuri and Meklin, 2006) which reduce the accountability structure of the organization. Robnson (2003) refers the little training of government managers in the private business model as a factor that difficult to make them accountable for achieved goals.
The decoupling between performance indicators and the setting goals and targets associated is another problem focused by prior research (Johnsen, 2005; Modell, 2005; per Laegreid et al., 2006) which cause dysfunctions on the feedback provided about achievements and results and reduce the useful of information for control and decision-‐
making (Modell, 2005). In addition, the lack of stability of performance indicators used over time difficult analysis and comparisons of performance which is one of the purposes of PM practices.
Despite these dysfunctional consequences that can be typical in public sector organizations, we argue in this paper that the importance attached to PM practices will be influenced by the degree to which agencies have adopted new managerial and accountable structures. So, we expect that government agencies more oriented to NPM reforms are more willing to use PM practices because they have an accountability and internal management structure more favourable in these contexts. We also agree with Poister and Streib (1999) that managers rather than politicians are the main users of performance information. Therefore, the first research question of this paper is concerned with the relationship between the implementation of NPM reforms and the use of PM practices by top managers of government agencies:
1. Is the level of implementation of NPM reforms positively associated with the use of performance management practices?
In this paper we are also interested in assessing the performance consequences when management makes adaptations to their internal structure. After the ‘agencification’ process the increase of management autonomy has been incentivized as a way to increase the effectiveness and efficiency of public entities and to make them more oriented to results and achievements (Osborne and Gaebler, 1993; Hood, 1991 and 1995). This ‘agencification´
process imply that executives/managers became more directly and clearly accountable in terms of their ability to produce results and to run their organizations efficiently (Osborne and Gaebler, 1993; Hood, 1991 and 1995). Therefore, the basic postulate is that decentralization and power delegation to units/agencies should help increase the responsiveness, effectiveness and efficiency of the public sector (Eckardt, 2008). So, contrary to Abernethy and Lillis (2001), we believe that differences on internal management structures will have consequences on the organizational outcomes (direct relation). In fact, as a consequence of the improvement of the accountability and the internal management structure, an improvement in the organizational performance is expected which leads to the second research question:
2. Is the level of implementation of NPM reforms positively associated with the organizational performance?
On the other hand we argue that using PM practices (this is, set goals and measure whether they are achieved), organizations reduce and eliminate ambiguity and confusion about objectives (Verbeeten, 2008; Walker et al., 2011) helping them in pursuit their mission.
Several prior studies find empirical evidence about the likely success of target setting and PM practices in improving public service performance (see for example, Hyndman and Eden, 2001; Hendrick, 2003; Cavaluzzo and Ittner, 2004; Verbeeten, 2008; Walker et al., 2011). In this line, prior studies show increasing evidence about the positive effects of PM on public sector performance (Moynihan, 2008; van Dooren et al., 2010; Walker et al., 2010). Also Wynen and Verhoest (2012) find that the use of PM techniques is a crucial step in enhancing the organizational performance.
In addition, Walker et al. (2011) reinforces the need to have appropriate structures of delegation and control to ensure that goals are implemented and targets are achieved and,
consequently, the organizational effectiveness is happened. Also Abernethy and Lillis (2001) find significant interdependencies between structure, the use of PM systems and the organizational performance. So, if the use of PM practices in public sector organizations can improve the performance of those organizations, we argue in this paper that this relationship is dependent upon from the orientation to NPM postulates (accountability and internal management structure). This argument leads to our third research question about the influence of NPM reforms in the relationship between the use of PM practices and the organizational performance:
3. Is the relationship between the use of PM practices and the organizational performance dependent upon from the orientation to NPM reforms?
3. Research design
3.1 Method and sample characterization
Using a bi-‐variate and multivariate analysis, we study the relationships between the orientation to NPM postulates, the use of PM practices and the organizational performance.
We provide empirical evidence from Portuguese government agencies based on the survey method. Based on the unique survey data obtained from Portuguese government agencies,1 we study the use of PM practices (goals-‐setting and targets, use of appropriate performance indicators and the stability of performance indicators) in relation to their context (NPM postulates), as well as their relationships with the organizational performance. The unit of analysis is each responding agency. Generally, this was done by the Chief Executive Officer (CEO), but it was possible to delegate the respondent function to other persons that are knowledgeable about the organization as a whole (planning and control mechanisms, implemented PM systems, autonomy and cultural issues and organizational performance).2 The majority of respondents are CEOs (38%) or senior managers (44%) of government agencies (assistants, vice-‐presidents, directors, sub-‐directors, coordinators) who are very familiar with the high-‐level decision-‐making process. Their work experience is, on average, of about ten years. About 99% have a graduate degree and 20 respondents have PhDs. The
1 Our survey was adapted from the COBRA survey (Comparative Public Organization Data Base for Research and Analysis) to the context of Portuguese agencies. Through the Centre for Research in Public Policy and Administration (NEAPP), Portugal is a partner of this academic research network in the field of public management since March 2007. The final version of the Portuguese survey was obtained after incorporating the feedback given by four CEOs, who did the pilot tests.
2 Under the organizational theory, individual top managers are the appropriate unit of analysis on this kind of study because perceptions, willingness, beliefs and behaviors towards a particular change and innovation are determined by each one within the organization (Anderson and Young, 1999).
survey data is the only data available on attitudes, perceptions, and behaviors in Portuguese public sector organizations.
The survey was available online between October 2009 and May 2010. Each agency was contacted by post and e-‐mail, whereby we presented the objectives of the study, the way of collaboration and the indication of the access link where they should access the survey.
Several calls and e-‐mails were answered in order to clarify questions and doubts regarding the survey. In January 2010, a second request was sent by post and e-‐mail to the agencies.3 The questionnaire was sent to a population of 342 agencies distributed by different ministries (see Table 1). Of these, 155 questionnaires were filled out for a total response rate of 45%.
Insert Table 1
Our sample consists of two categories of agencies: (0) legally independent agencies (114 agencies) and (1) dependent agencies (41).4 Following the framework of the COBRA database (on which we base our questionnaire), we focus on the steering relationship between agencies that report to government/minister/department and the oversight authority as a complete result of the steering cycle. Subordinate agencies/subsidiaries that report to another agency were not included in our sample because they are not comparable to other agencies covered on our study (COBRA document, 2010).5 Agencies that report directly to an oversight authority are more involved in the processes of planning, control and performance measurement, and are more accountable for results and goal achievements than subordinate agencies. Moreover, using the same assumptions of the COBRA partners we assure the comparability of our data with the data provided by agencies of other countries that applied the same survey.
Table 2 presents the sample characterization by each one of the two categories of agency, focusing on size (by full time employees and budgeting) and self-‐funding (self-‐generated income is the main source of income). Data show that, on average, legally independent agencies are larger organizations (considering both FTE and budgeting) and have more
3 We do not find significant differences between previous responses and those from the second request.
4 These are agencies that operate closely to the respective parent ministry (also called departments) and consist of central (national) units to which many regional or local offices report (e.g. general secretaries, general directorates; general inspections, etc.). On the other hand, legally independent agencies are single organizations without subordinate units that cover the territory under review (e.g. Universities, Hospitals, public institutes, etc.). They are structurally differentiated from the State and have some capacity for autonomous decision-‐
making.
5 Available on http://soc.kuleuven.be/io/cost/survey/index.htm.
financial independence (self-‐funding) in relation to the oversight authority than dependent agencies. Differences on size are statistically significant (p-‐value < 0.01).
Insert Table 2 3.2 Measuring NPM vs non-‐NPM agencies
Hood (1995) discusses the rise of NPM as an alternative to the tradition of public accountability embodied in progressive era of public administration ideas. In summary, he identifies the following NPM doctrines: (1) make the units manageable; (2) introduce the notion of competitive markets and term contracts; (3) adopt private sector management practices; (4) rationalize the use of resources (value-‐for-‐money); (5) clarify assignments of responsibility (accountability) and (6) make explicit performance measures.
Following these doctrines, several arrangements in the internal organizational structure have been introduced in public sector organizations, particularly the increase of the division/decentralization of the units after the NPM paradigm, resulting in accounting changes (Lapsley and Pallott, 2000; ter Bogt and van Helden, 2000) and in a greater need for more quantitative and sophisticated performance measurement and control systems (ter Bogt, 2003 and 2004; Budding, 2004). On the other hand, several authors refer the financing structure of the organizations as an important factor in the increase of efficiency and performance (van Helden, 2000; Geiger and Ittner, 1996; Budding, 2004; Cavaluzzo and Ittner, 2004; Wynen and Verhoest, 2012). As said by Wynen and Verhoest (2012:10), for public sector organizations which incomes result predominantly from self-‐generated incomes ‘it is very important to monitor very well the quantity and quality of services produced and delivered, and the efficiency of their operations, as these elements will immediately impact upon their financial viability’. In other words, the more the entities that depend on self-‐generated incomes, the greater the uncertainty of the financing, which can encourage the use of more sophisticated control systems. In this way, literature identifies a set of aspects of internal organizational structure that determine the use of accounting innovations (like PM practices), as well as its impact in the organizational performance.
In this section, we tried to characterize the agencies of the sample according to its orientation to NPM reforms. The objective is to obtain two groups of agencies: 1-‐ the NPM agencies (more orientated to NPM reforms), and 0 -‐ non-‐NPM agencies (less oriented to
NPM reforms). Based on NPM literature, we employ three survey questions for to measure this variable: (1) On what basis is the CEO of the organization accountable to the oversight authority for results and goals achievement? (2) To what extent do you agree with the following statements (about the internal organizational structure) in your organization as indications of performance? and (3) Is self-‐generated income the main source of income of your organization? The first two variables are measured in a likert scale of five points (1 – not at all; 5 – to a very great extent). The third indicator is a dichotomous variable that assumes a value of 1 if the self-‐generated income is the main income of the organization (meaning a greater uncertainty of funding) and value of 0 if not.
Table 3 presents descriptive statistics for each one of these indicators. The extension with which the CEO responds, before the parent Ministry, for the outcomes achieved is the indicator with more relevance (4.19). This extension tends to increase in the bigger agencies (higher number of employees). On the other hand, the management of units is made based on the outcomes (average of five items) in only 20% of the agencies,6 with high or very high extension. Finally, a great dependence of external financing is observed (namely of transfers from the State Budget), in that only 24% of the agencies reveal that their own income is their main financing source. This reality does not vary according to the size of the agencies.
Insert Table 3
In this paper, the NPM variable is a dichotomous variable that assumes the value 1 (NPM agencies) when at least two of the three following criteria are fulfilled: (1) the CEO is accountable to the oversight authority for the achievements, with high or very high extension; (2) in average, respondents agree with the survey items that feature the internal management structure, with high or very high extension; and (3) self-‐generated income represents the main source of income. It assumes the value 0 if not. Thus, it can be stated that the sample is comprised by 49 NPM agencies and 106 non-‐NPM agencies. Associating this variable with the category of each agency (0 – dependent agencies; 1 – legally independent agencies in relation to the parent ministry), it can be stated that the majority of the NPM agencies belong to the Indirect Administration (greater independence from the
6 The outcomes of the factor analysis show that the five indicators have factor loadings higher than 0.5, constituting a unique factor with an alpha of 0.7051.
parent ministry), as presented in Table 4. The result of the statistical test Pearson chi2 demonstrates that this association is statistically significant (p-‐value < 0.01).
Insert Table 4
The descriptive statistics of the variables presented in the following sections are analysed considering the great or less orientation of the agencies to the NPM (1-‐ NPM agencies; 0 non-‐NPM agencies). To test the differences between the average of both samples, the Mann-‐Whitney U-‐test was applied.7
4. The use of PM practices by NPM agencies
In this section we explore the first research question of the paper. Several questions were addressed in order to understand the use of PM practices in Portuguese government agencies. The process of goal setting and specification, the use of appropriate performance indicators to measure achievements and stability of performance indicators represent the most important variables in this study about the use of PM practices. The measuring and descriptive statistics of these variables by NPM agencies are presented below.
4.1 Goals setting and specification
The process of goal-‐setting can be more or less participative, depending on the influence of the parent ministry. The questionnaire includes a question that aims to understand this process in the Portuguese public agencies. This variable may assume values between 1 and 5, depending if (1) the parent ministry establishes the objectives without consulting the organisation; (2) the parent ministry establishes the objective, after consulting the organisation; (3) the organisation establishes the objectives together with the parent ministry, through a negotiating process; (4) the organisation establishes the objectives after consulting the parent ministry or (5) the organisation establishes the objectives alone without interference of the parent ministry.
On the other hand, another question was addressed about the extent to which the objectives are specified in some document with measurable targets associated (in a scale increasing from 1 to 3).8 In fact, a central issue for a correct measurement and evaluation of
7 The Mann-‐Whitney U-‐test is appropriate to test the interdependence between the two samples when the data do not follow a normal distribution (Gibbons & Chakraborti, 2003).
8 Wherein 1 -‐ the objectives are not specified; 2 – objectives are specified but without targets associated; and 3 – the objectives are specified with targets associated.
achievements is the association of measurable targets to the objectives (Otley, 1999;
Poister, 2003; Tuomela, 2005; Broadbent and Laughlin, 2009; Ferreira and Oltey, 2009), although this can be really problematic in some public services where the produced outputs are not easy to measure (Feltham and Xie, 1994; Ittner and Larcker, 1998; Modell, 2001;
Dooren, 2005; ter Bogt, 2008; Verbeeten, 2008 and Arnaboldi and Azzone, 2010).
Table 5 presents the descriptive statistics, as well as the measurement of these variables.
The outcomes show that, on average, the process goal-‐setting results from a negotiating process between the agency and the parent Ministry (3.45), which demonstrates a favourable situation. The differences regarding the compliance to NPM features are not significant (3.35 for non-‐NPM agencies and 3.65 for NPM agencies). A possible justification of this similarity between the two groups can be related to the fact that the current legislation foresees a participation process between agencies and the parent ministry in the definition of the objectives (for example through the SIADAP – the Integrated Performance Assessment System for Public Administration).9 So, legal pressures can be associated with this result.
Regarding the objectives specification level we see that, on average, they are specified in a document, but usually without targets associated (2.05). There are significant differences between NPM and Non-‐NPM agencies (p-‐value < 0.01). This means that it is among the NPM agencies that the specification of the objectives with associated targets becomes more common. In addition, the most used documents for this purpose are internal documents (2.32) following by documents focusing on individual objectives which mean that agencies have preference for internal documentation as a way to specify goals and the associated targets. The use of this practice is greater among NPM agencies, especially regarding the use of documents for internal purposes and the budget allocation document.
Insert Table 5 4.2 Measurement of the achieved performance
Performance measurement frameworks become less accounting-‐oriented and move away from to evaluate subunits performance (Abernethy and Lillis, 2001) and manage strategic uncertainties (Simons, 1995). As said by Walker et al. (2011: 372), PM systems are needed
9 Approved by the Law n.º 66-‐B/2007 of 28 December.
‘because public organizations cannot plan for performance improvements if they are not aware of the starting point and do not have information for manager’s to measure progress toward targets’. In public sector, performance measures are referred primarily to those
‘performance indicators of efficiency, effectiveness and equity that are intended to be used to improve rational decision-‐making in administrative and political processes’ (Johnsen, 2005: 9).
In this paper, the extension to which agencies use performance indicators is measured based on the instrument used by Cavaluzzo and Ittner (2004) to assess performance measurement systems. Respondents were asked, in a five-‐point Likert scale (from 1-‐ not at all to 5-‐ very great extent) about the extent to which organizations use performance indicators that measure (1) quantity of outputs, (2) quality of service delivery, (3) use of resources, (4) effectiveness, (5) efficiency, and, (6) societal effects (see Table 6).
On average, the agencies use the performance indicators with a reasonable extension (3.73), highlighting the indicators that measure the quantity of outputs (4.17). Data shows that indicators of quantity and quality of outputs, use of resources and efficiency used to a greater extent (4.17, 3.88, 3.87, and 3.83, respectively); this is an expected result, considering the Portuguese governmental recommendations introduced by the SIADAP that obligate to measure resources employed, efficiency, effectiveness and quality of outputs. In opposition, societal effects are not well used by respondents (2.92) confirming the pessimistic opinion of Modell (2005) about the little use of these kind of measures. These results suggest that our findings are consistent with the theory regarding the difficulties to measure and assess outcomes and effects of government work in the society (see, for example, Kaplan, 2001; Collier, 2001; Modell, 2001; Dooren, 2005; Vakkuri and Meklin, 2006).
With respect to the differences between NPM and non-‐NPM agencies in the use of PM practices, we see that NPM agencies use the performance indicators with higher extension, in comparison with non-‐NPM agencies. These differences are statistically significant (p-‐value
< 0.01). This positive association between the orientation to NPM postulates and the use of the performance indicators is based on the assumptions of the economic theory. In other words, the selection of the most sophisticated control systems, through the use of performance measures, depends on the adaptation of the internal structure of agencies to
the NPM doctrines (ter Bogt and van Helden, 2000; Budding, 2004; Johnsen, 2005; Arnaboldi and Azzone, 2010; Wynen and Verhoest, 2012).
Insert Table 6
To complement this analysis, we address more two survey question that measure (1) the extent with which these indicators correctly measure the organisational activity, as well as (2) the stability of the indicators in the last five years. Data show that, in the respondents opinion, indicators are used correctly to measure the organisational activity with very high extension (2.64), and there has been reasonable stability throughout the last five years (2.24). As we expect, the NPM agencies demonstrate a more favourable situation in both situations, which indicates a higher sensitivity of these agencies to measure and monitor performance. These differences are statistically significant (p-‐value < 0.01). This way, in this section we confirm our expectations about the positive impact of NPM reforms in the use of PM practices (first research question).
5. Effects on the organizational performance
5.1 The organizational performance by NPM agencies
In this section we try to answer to the second research question about the relationship between NPM agencies and the organizational performance. The need to satisfy multiple objectives turns performance into a multidimensional concept for which no single overriding measure is adequate (Feltham and Xie, 1994; Ballantine et al., 1998; Brignall and Modell, 2000; Bispe et al., 2007; Otley, 2008, cited by Ferreira and Otley, 2009; Walker et al., 2011).
Different approaches of performance were found on the literature that distinguishes the focus (Verbeeten, 2008) and the quantification of performance (Feltham and Xie, 1994;
Ittner and Larcker, 1998; Banker et al., 2000; Reck, 2001). Verbeeten (2008) distinguishes between quantitative and qualitative performance. Quantitative performance is more concerned about results on the efficiency, effectiveness and the quantity of outputs, while qualitative performance involves societal effects (outcomes) of agencies’ activities (e.g.
levels of citizen satisfaction) and the quality of delivered services; so, qualitative information is more useful to inform about long-‐term performance (Kloot and Martin, 2000; Kaplan, 2001; Ittner and Larcker, 2003; Widener, 2006) while quantitative information is
characterized by a short-‐term perspective (Said et al., 2003). For an adequate performance evaluation, a multidimensional approach should be used based on these two perspectives.
In this paper, two survey questions were employed to measure the organizational public performance: one single measure that measures the respondents’ self-‐assessment related to general results of the organization; and the performance rating based on eleven different criteria: (1) efficiency; (2) effectiveness; (3) quality of service delivery; (4) motivation; (5) satisfaction of staff; (6) quality of management; (7) internal cohesion; (8) flexibility of the organization; (9) stability of the organization in the environment; (10) responsiveness to society; and (11) accountability towards society. Table 8 presents the main descriptive statistics for these variables (a ten-‐point likert scale were used: 1= poor performance; 10 – very good performance). This way, the organizational performance is measured by a single measure of performance and a multi-‐dimensional approach of performance that assesses goals achievement (effectiveness, efficiency and quality), employee satisfaction and internal environment (motivation and satisfaction), as well as the impact on the society (accountability and responsiveness). Descriptive statistics on the overall performance, as well as the average performance resulting from the eleven criteria of performance are presented in Table 8.
Insert Table 8
Similar to the findings of Verbeeten (2008), our data show that, on average, the respondents perceive the overall organizational performance as being quite good (7.72). This result is not so different from results when we measure performance in different criteria. However, we highlight the performance at the level of the accountability and responsibility to society (8.21 and 7.9, respectively), stability of the organisation (7.99), as well as the effectiveness (7.99) and the quality of the services provided (7.97), as these criteria are in the top of the ranking. Analysing the comparison between the two groups of agencies, the results show statistically significant differences in several indicators of performance, particularly in the overall performance and the performance at the level of the accountability and responsibility to society (p-‐value < 0,01). This means that the respondents of NPM agencies perceive a better performance of the organisation in several aspects, as compared to the respondents of non-‐NPM agencies. Apparently, the agencies more oriented to the NPM (a context that favours accountability and an internal management oriented to units) will
perform better than non-‐NPM agencies, which is consistent with the basic principles of this paradigm (Hood, 1995). These results are coherent with our second research question which reinforces theory about NPM effects on public performance (Osborne and Gaebler, 1993;
Hood, 1991 and 1995; Eckardt, 2008).
5.2 Influence of PM practices on the organizational performance by NPM agencies
In this section we explore the third research question of the paper. For to test this relationship we use an ordered logit regression.10 The purpose is to explore the positive relationship between the use of PM practices and the organizational performance as well as the differences associated with the orientation to NPM. Based on prior studies that investigate the effects of PM systems on the organizational performance (see for example, Abernethy and Lillis, 2001; Cavaluzzo and Ittner, Verbeeten, 2008 and Walker et al., 2011) we introduce in the model five variables presented in section five: (1) the goals setting process (GOALS_set), (2) the specification of goals with targets associated (GOALS_specification), (3) the average use of a range of performance indicators (INDICATORS_use), (4) the extent to which performance indicators are appropriate (INDICATORS_grasp) and, finally, (5) the stability of indicators (INDICATORS_stability). For the dependent variable we employ an overall measure of performance (OVERALL performance) and the average performance assessment in a range of the eleven criteria presented above (AVERAGE performance). For each model we explore the influence of PM practices on performance (full sample) as well as the differences regarding subsamples of NPM agencies and non-‐NPM agencies.
Table 9 presents the estimated coefficients for each independent variable, with z-‐statistics in parenthesis. The full sample has 144 observations in model 1 (OVERALL performance) and 146 observations in model 2 (AVERAGE performance). For the regression results we tested collinearity using the Variance Inflation Factor (VIF). No VIF-‐value higher than 1.10 was reported in the models indicating that correlations between independent variables do not cause a problem of multicollinearity.
10 The use of OLS is problematic in estimating the determinants on ordinal variables because the assumptions of OLS are violated when it is used with a non-interval outcome variable. Given that our dependent variables are ordinal variables (i.e., the categories are ordinal), the most appropriate regression technique is ordered logit regression (OLOGIT) (Hosmer and Lemeshow, 2000; Gibbons and Chakraborti, 2003).
The likelihood ratio chi-‐square of 28.30 and 24.20, respectively for models 1 and 2, with p-‐
values of 0.000 tells us that our models as a whole are statistically significant, as compared to the null model with no predictors. On the other hand, the larger pseudo R2 for the subsample of NPM agencies, in both models (overall and average performance), shows a better fit between the use of PM practices and the organizational performance in NPM agencies, comparatively to the full sample and to the subsample of non-‐NPM agencies.
Generally speaking, these statistical results reveal that a good part of the variation in the organizational performance is explained by estimated models; in addition, a better fit was finding for NPM agencies (both models) meaning that improvements in performance as a result of the use of PM practices will be greater if agencies adequate the PM system to their structures. This confirms our expectations about the importance to adjust the use of PM practices to changes in the organizational structure for better effects on performance (Abernethy and Lillis, 2001; Gerdin and Greve, 2008; Bispe et al.2007; Walker et al., 2011).
Analyzing the estimated coefficients, we find that the stability of indicators represents the most robust determinant in both models, for the full sample and the subsample of NPM agencies. This means that more stable indicators are more appropriate to monitor and control actions which lead to greater improvements in performance. This effect is greater in NPM agencies (p-‐value <0.01). Also the use of a range of performance indicators and the extent to which they grasp real events with regards to the activities of the organization are statistically significant in the first model. These results reinforce prior studies that find a positive effect of PM systems on the organizational performance (Abernethy and Lillis, 2001;
Hyndman and Eden, 2001; Cavaluzzo and Itner, 2004; per Laegreid et al., 2006; Verbeeten, 2008; Walker et al. 2011). However, no significant results were finding about the influence of the goals setting process and its specification with targets associated on performance (for both models), contrary to conclusions of Verbeeten (2008). To sum, despite the unintended consequences of NPM reforms referred by some scholars (Brunsson and Olsen, 1993; Groot and Budding, 2008; Lapsley 2008 and 2009) who make problematic held managers accountable in non-‐profit organizations and monitor and control performance, our results help clarify the role of changes introduced by NPM reforms on public performance.
Insert Table 9 6. Discussion and main conclusions
Based on assumptions of the NPM paradigm, we investigate whether the use of PM practices (first research question) and the organizational performance (second research question) are associated with changes introduced by Portuguese government agencies in their structures (associated with accountability and internal management structures). In addition we explore in this paper the influence of adjustments made on internal structures (orientation to NPM postulates) on the fit between the use of PM practices and the organizational performance (third research question). The employed bi-‐variate and multivariate analysis shows empirical evidence that confirms our expectations.
First of all, our data shows that Portuguese government agencies are yet highly dependent from the parent ministry and less oriented to NPM postulates like the adoption of more accountable and autonomous structures, resulting in 49 NPM agencies and 106 non-‐NPM agencies. Our findings suggest that agencies that made structural arrangements as a way to improve efficiency and effectiveness of public government actions (NPM agencies) are more willing to use PM practices, especially in goals setting with targets associated and in measurement of achieved performance. Secondly, we also find empirical evidence on the positive relationship between NPM agencies and the organizational performance. So, despite some mixed results finding in prior studies on this relationship, this research contributes to reinforce a positive relationship.
Thirdly, this paper adds to knowledge about the role of NPM reforms to explain the relationship between the use PM practices and the organizational performance. In fact, we find a positive relationship between the use of PM practices and the organizational performance (stronger evidence for the use of performance indicators and its stability) which is consistent with prior studies that associate enhancements on performance with the use of PM practices (e.g. Abernethy and Lillis, 2001; Hyndman and Eden, 2001; Cavaluzzo and Ittner, 2004; per Laegreid et al., 2006; Verbeeten, 2008; Walker et al., 2011). However, our findings suggest that this positive and significant relationship is dependent upon from the agencies capacity to adopt new structural arrangements under the NPM reforms. We think this is a practical contribution of this paper because these results help to clarify the way the organizations should adapt their structures for to be more effective in the use of PM practices. In our opinion we can say that our findings provide enough evidence to confirm expectations of the three research questions explored in this paper.