http://dx.doi.org/10.1590/0104-530X2512-16
Resumo: A intervenção em um fenômeno como um Sistema Nacional de Inovação (SNI) é algo complexo, principalmente em economias em desenvolvimento. Estudos constatam a ausência de instrumentos e ações práticas que permitam atuar no aprimoramento contínuo desses sistemas. No Brasil, alvo do estudo, há um caso de ação pioneira desenvolvida pela EMBRAPII – Empresa Brasileira de Pesquisa e Inovação Industrial, que publicou recentemente um padrão de sistema de gestão baseado em processos de negócio que serve de orientação para apoiar a gestão das instituições de pesquisa cientíica e tecnológica (ICT) parceiras da instituição. O artigo apresenta uma proposta de modelo de avaliação como mecanismo para a melhoria de uma rede de ICTs que utiliza o padrão de sistema de gestão, visando avanços na gestão da inovação e impactos positivos no SNI. O modelo está estruturado com base na integração de três conceitos principais: sistema de medição de desempenho, processos de avaliação, auditorias internas e externas. As limitações e hipóteses sobre a utilização de um modelo como esse são discutidas. As principais conclusões são que não há modelos similares na literatura e que a aplicação desse modelo pode ser um instrumento importante para países em desenvolvimento que precisam dinamizar sua rede de inovação. Palavras-chave: Sistemas Nacionais de Inovação; Instituições de pesquisa; Inovação; Avaliação de redes de inovação; Auditoria interna; Auditoria externa.
Abstract: The interference in a phenomenon as a National Innovation System (NIS) is complex, especially in developing economies. Some studies discuss the absence of tools and practical actions that allow a continuous improvement of these systems. In Brazil, there is a case of pioneering action developed by EMBRAPII, an agency for Industrial Research and Innovation. EMBRAPII has recently published a standard of management system based on business processes. That tool aims to provide guidance to support the management of R&D Institutes, partners of EMBRAPII. This paper proposes a model of evaluation as a mechanism to improve a network of R&D Institutes using that standard management system. The model is structured based on the integration of three key concepts: performance measurement system, evaluation processes, internal and external audits. The constraints and assumptions about the use of a model such as this are discussed. The main conclusion is that there are no similar models in literature and application of this model can be an important tool for developing countries that need to boost their innovation network.
Keywords: National Innovation Systems; R&D institutes; Innovation; Evaluation of innovation networks; Internal audit; External audit.
Evaluation of innovation networks based on
standardized management system
Avaliação de redes de instituições de pesquisa científica e tecnológica baseada em um sistema de gestão padronizado
Silvia Ronsom1 Daniel Capaldo Amaral1
1 Departamento de Engenharia de Produção – EESC, Universidade de São Paulo – USP, São Carlos, SP, Brazil, e-mail: sronsom@gmail.com; amaral@sc.usp.br
Received June 30, 2015 - Accepted Dec. 09, 2015 Financial support: CNPq and EMBRAPII.
1 Introduction
Governments from all over the world consider National Innovation Systems (NIS) essential to increase the innovative potential of countries. More than a decade ago, Lundvall et al. (2002) recognized increased interest on National Innovation Systems’ perspective and networks in Latin America, Asia and Africa.
R&D Institutes play a signiicant role on the
innovation scenario of a country and its evolution
relies on a set of technical and organizational skills. The Oslo Manual (OCDE, 2005) points out
is also scientiic and technology institutions that feed commercial innovation, providing scientiic
Aligned with Lundvall et al. (2002) and Oslo Manual (OCDE, 2005), it is possible understand that one way to improve the NIS is directly related to operational improvement and expansion of capacity in providing services by R&D Institutes. The NIS, in these cases, shall improve itself on abilities such as: 1) plan technological strategies; 2) plan compatible business that support its actions; 3) manage portfolio of insights (demand) and its projects; 4) establish business processes which allow operational improvement; 5) develop its skills constantly.
For Viotti (2002), there are relevant differences among developed and under developed countries’ NIS. Liu et al (2014) made a comparative research in which the Brazilian NIS was suited on a group of emerging countries characterized by a strong focus on the process of knowledge production and failure
on business beneit acquisition.
Arbix et al. (2005) showed that cooperation with R&D Institutes is an important feature for the Brazilian NIS development. On a study about exporting and innovative companies, with data from the main domestic foundations (PINTEC, SECEX and RAIS), it was detected among the Brazilian exporting companies, that released technological innovation, a higher concentration of companies with interest on cooperative arrangements and government investment, including R&D Institutes participation.
Assad & Souza (2009), in speciic case study, validate
the importance of R&D Institutes involvement on
the innovation of medical ield.
Another specific characteristic of Brazilian NIS is that its potential could be more exploited if considered data showing several challenges faced by R&D Institutes when supporting the productive system. There are structural issues such as the legal framework regarding payment of services provided and foundations engagement (Jannuzzi et al., 2008),
lack of clear deinition of the researchers’ role
(Assad & Souza, 2009) and intellectual property issues (Kamimura & Cornetta, 2011) that discourage investment and partnership. And managerial issues such as the need for a more professional operation and improvement of laboratories best practices and management models, as argued by Assad & Souza
(2009) on medical ield, and Oliveira & Telles (2011)
on several industrial sectors.
The challenge then is how to spread management models and establish good practices, including techniques for prospecting, planning and developing technologies and products in R&D Institutes network. According to Chandler (1962), Penrose (2006), Rutman (1977), Bin (2008) and Chiesa et al. (1996), the evaluation of these innovation networks has a fundamental role.
A chance to investigate this issue is the recent case of the Brazilian Agency for Industrial Research and Innovation (EMBRAPII). The strategic plan related to industrial innovation established by Federal
Government comprised among the initiatives, the foundation of EMBRAPII as an institution aiming to acting particularly on the expanding R&D Institutes activities. EMBRAPII was founded to accredit and manage a network of R&D Institutes able to prospect and develop innovative technologies, in association with the productive sector. Based on the differential of other similar networks as the Fraunhofer Institute in Germany and Carnot in France, EMBRAPII
developed a standard of Management System speciic
to its network needs, called EMBRAPII Operational Excellence System (EOE System). The purpose according to the standard is to guide business processes of the EMBRAPII network and to be reference to establish a common language among the stakeholders
in order to support EMBRAPII on qualiication and
monitoring accredited R&D Institutes. The EOE System is available at EMBRAPII website.
This article proposes an evaluation model for R&D Institutes network based on the EOE System. The proposal is designed to suit in particular to the Brazilian NIS, represented in this study by EMBRAPII innovation networks institutions. The main contribution conducted by an action-research is the model introduction and the discussion about its components and aspects that affect the development and evolution of practical instruments related to the improvement of EMBRAPII’s innovation network.
This paper is organized in six sections. The bibliographic review is presented on the following section. Section 3 contextualizes the case studied (and the research issue). On section 4 is explained and analyzed the proposed model, including notes about the research outline by means of the knowledge
acquired on the literature. Section 5 contains inal
considerations and conclusions made by the research team.
2 Bibliographic review
2.1 Evaluation of innovation networks
The agents of a National Innovation System (NIS) and its type of performance and relationship are quite different among countries and regions. The OECD (2009) mentions some principles that should be applied by those who work with innovation, such as seeking and arranging knowledge in a globally way, integrating cooperative networks and compounding partnership, and handle the needs of developing countries (OECD, 2009).
developed countries. Furthermore, Lundvall et al. (2002) indicate absence of concepts, instruments and practical actions that allow continuous improvement of NIS. Levén et al. (2014) also points out that is essential to have a common mechanism for those involved on the network, to establish, promote and deploy processes in an innovation network. This article is focused on the absence of instruments and practical actions that allow continuous improvement of NIS.
Nobelius (2004) suggests that one of the main
dificulties for innovation management is turning
research works into commercial products, such as turning
creation into innovation. Some of these dificulties
derive from internal and external communication problems, cultural barrier, use of improper processes and methods not necessarily structured by superior
and mature technologies. Such dificulties require an eficient management of innovation process
(Nobelius, 2004; Schulz et al., 2000).
In the case of Brazil, recently, Liu et al. (2014),
while studying the eficacy perspective on innovation
systems, pointed out the country inside a group of emerging countries characterized by a strong focus on knowledge production process, along with India, Mexico, Thailand and Malaysia. The authors highlighted the group capacity to manage costs with R&D to uphold a reasonable productivity level on industry and services. According to the authors, despite being strong on knowledge production, these countries do not show the same strength on application and
acquisition of business beneit. Oliveira & Telles (2011)
based on previous experiment, highlighted that it is necessary that the R&D institutions are recognized by their technical skills on the project management
ield and be highly eficient on its management as a
whole. The aspects presented along with EMBRAPII study opportunity led the research team to consider Brazil as a country target of study.
Chandler (1962) and Penrose (2006) characterize the management activity as a set of decisions related to coordination, evaluation and planning of the company’s work and allocation of its resources.
In this context, evaluation is, irst of all, a process that applies scientiic procedures in order to gather
reliable and valid evidences about how and to what
extent certain activities generates certain effects
or results (Rutman, 1977). Among the beneits of an evaluation stands out priority identiication and
provision of structural conditions to support activities conducted by the agents that compound an innovation system (Bin, 2008).
Lengrand (2006) discusses about four evaluation types intended to innovation programs: 1) Ex-ante evaluation: taken prior to implementation; 2) Midterm evaluation: review of progress or results at some point of their development; 3) Real time evaluation: detailed monitoring in the execution; 4) Ex-post evaluation: analyze results after being carried out. This typology was later accepted for creating the evaluation model of EMBRAPII innovation network.
Brazilian studies on Science and Technology (S&T) evaluation have been conducted in a comprehensive way. As in Salles-Filho et al. (2010), that show a study about impact evaluation in innovation programs. Bin et al. (2003) and Paulino et al. (2007) study technological programs evaluation for environmental impact. Furtado et al. (2002), explore S&T indicators for evaluation of university and industry cooperation programs. They discuss the networks evaluation in a broad way, but do not present instruments that guides how to improve each aspect of the organization, as the standards of operation and business processes of innovation network, for instance. Therefore, the main issue would be how to establish and promote an operational evaluation model that guides R&D Institutes in developing countries, ensuring that their activities are in accordance with the common goal of the innovation network they belong to. Such model should allow a detailed monitoring of R&D Institutes operation.
The overview about evaluation principles of R&D Institutes presented by Zackiewicz (2005) assisted the development of this study on elaborating an evaluation model for research institutions. On the Chart 1 is presented some comments of the authors related to the selected literature notes.
As previously mentioned, managers of research
institutions face a challenge. It means inding ways
to disseminate and use standards, such as the EOE standard, to improve R&D Institutes network they
Chart 1. Principles of S&T evaluation considered by the author.
Zackiewicz (2005) Author’s comments
To evaluate is to interpret an attribute by means of a criterion
This proposal suggests the use of EOE standard as the main criterion for R&D Institutes to perform an evaluation of its business processes and its operation as a whole. The EOE standard will be presented on section 3.
To decide is to take an action from an evaluation
This proposal suggests three evaluation aspects to support decision making on institutions: performance measurement system and internal and external evaluation. These will be presented and debated on subsections 4.1 and 4.3.
All evaluation methods are social constructions
assist or have interest on developing, combining the different instruments of networks evaluation for this purpose.
Therefore, it is necessary to know the most common evaluation models, which shall be presented hereinafter.
2.2 Performance measurement system
The development of the Performance Measurement System (PMS) remains a topic of increasing concern for both the Academy and Industry (Nudurupati et al., 2011, p. 280; Valmohammadi & Servati, 2011, p. 494; Neely et al., 2005, p. 1228). In the Academy, the performance measurement is widely discussed with the proposition of PMS new models and its development methods. However, many authors use different terms for the same concepts. The concept of performance measurement, for instance, has
different deinitions for Neely et al. (2005, p.1229),
Bourne et al. (2002), Atkinson (2000), Bititci et al. (2000), Kaplan & Norton (1997).
Therefore, is important to present some concepts
for a better understanding of the topic. The deinitions
adopted in this paper are in accordance with Neely et al. (2005, p. 1229):
• Performance measurement. It can be deined
as the process of measuring the eficiency and
effectiveness of an action.
• Performance indicator (Originally, Neely et al. (2005, p. 1229) use the term performance measurement, however it was adopted performance
indicator, most common word on the ield). It can be deined as a metric used for measuring the eficiency and/or effectiveness of an action.
• Performance Measurement System (PMS).
It can be deined as a set of performance indicators used for measuring both eficiency
and effectiveness of actions.
According to Gerolamo (2003, p. 59), among the PMS models proposed between the 1980s and 1990s the most researched and cited models were Performance Pyramid, Cross & Lynch, (1989); Balanced Scorecard (BSC), Kaplan & Norton, (1997); and Performance Prism, Neely et al. (2002).
In the speciic case of performance evaluation for
productive arrangements, operating on cooperation networks it can be cited Cardoza Galdamez (2007) who studied clusters of small and medium enterprises. Hansen & Oliveira (2009), who developed a model for systematic evaluation of competitive performance of productive arrangements. It is also highlighted by Lenz’s (2007), which studies about horizontal cooperation networks. According to Lenz (2007) the Prism and the Balanced Scorecard models of
performance may be basis for adjustment of the measuring performance of individual companies for horizontal cooperation networks.
The purpose of this choice is related to the Prism model focused on obtaining stakeholders’ satisfaction, considered closer to the networks reality, in which the agents (network participants, government,
suppliers, and costumers) have signiicant role on its
operation, setting a table of wide and complex social relationships. However, the prism does not suggest a way of deploying prospects that take into account an easy usage on practice, such as the Balanced Scorecard. Due to these characteristics objectives, indicators, goals and initiatives of the model proposed by Lenz (2007) followed Kaplan & Norton (1997) recommendations.
Thus, the perspective of stakeholders’ satisfaction of Neely et al. model (2002) and objectives outcome of Kaplan & Norton (1997) were chosen to compose the model proposed by Lenz (2007) and it guides the outline of this proposal.
2.3 Audit in innovation networks
The performance measurement is useful but is only one part of the solution, according to Chiesa et al. (1996). For the authors, it is essential to understand how it emerged and to have continuous improvement, it is necessary to review not only innovation skills of an institution, but the processes through which innovations are developed and exploited.
Chiesa et al. (1996) state that performance auditing helps on identifying problems and needs, but does not explain why are gaps on current performance and
no action plan to ill the gaps is provided. As deined
in Brazilian rule NBR 19011: 2002 (ABNT, 2002), audit is a systematic, documented and independent process for gaining audit evidence and objectively evaluating it, in order to determine to which extent criteria of process are met and, according to the author, meets these needs.
Authors like Hammer (2007), developed speciic
frameworks of process auditing to assist executives on planning companies’ changes, make progress and identify barriers. Chiesa et al. (1996) show a framework for technical innovation management auditing. The model approaches management processes and organizational mechanisms over which innovation is performed and provide basis for a detailed auditing of current innovation practices.
To perform evaluation of a program, according to the Brazilian Association of Technical Standards – ABNT 17021-2007 (ABNT, 2007), three conditions are prerequisites and must be tested in advance:
1) a program clearly deined; 2) objectives and results clearly speciied; 3) an intelligent scheme
of causalities linking program to objectives and results. This work considers EMBRAPII context and its Management System standard – EOE System, objectives and strategical goals, to be presented later on sections 3 and 4.
ABNT 17021-2007 (ABNT, 2007) describes
concepts of management systems certiication, useful
for this work, and according to the standard can provide independent statement that the organization’s
management system: 1) is in accordance with speciied
requirements; 2) is able to reach its declared policies and respective objectives with coherence; 3) is effectively implemented. According to the standard, accordance
evaluation such as the certiication of a management system provides signiicance for the organization, its
customers and interested parties.
3 EMBRAPII and the context of
proposal application
3.1 EMBRAPII presentation
The Brazilian Agency for Industrial Research and Innovation – EMBRAPII – is a result of a RD&I program established by the Federal Government in
2013. It seeks to it the gaps of RD&I programs and
to consolidate itself as an agent of innovation with
high eficiency. Its goal is to register and manage a
network of R&D Institutes capable of prospecting and developing innovative technologies. The program is similar to those existing in countries like Germany with the Fraunhofer Institutes, France with the Carnot Institutes and England with Catapult Institutes.
One of the challenges faced by EMBRAPII was
the issue identiied by Liu et al. (2014) and Oliveira
& Telles (2011) related to the research institutions on the Brazilian NIS with low experience, regarding technology transfer. As mentioned on section 2, Brazil is characterized by a high production of “papers”, but few insertion and impact of this production on the commercial and industrial development. Therefore, besides composing the network, as in French and German cases, there is the extent that on Brazilian program is essential to incorporate the issue of developing institutes belonging to the network in order to impact NIS as a whole.
As mentioned by Oliveira & Telles (2011), cooperation experiences between university-company, from 2008, by the IPT (Instituto de Pesquisas Tecnológicas) focused efforts to identify relevant themes for innovation, to
encourage the creation of an ofice of projects and
planning of new technologies. However, still at pilot
phase, EMBRAPII faced the absence of a consolidated model capable of providing conceptual basis to guide operational activities for its Units, shown on section 2. Such absence took EMBRAPII, along with
a researcher group, to develop a speciic innovation
management system to its accredited Units, the R&D Institutes. The system was called EMBRAPII Operational Excellence System (EOE System) and is described as follows.
3.2 EOE system - general overview
The introduction stage of an innovation, capable of generating competitive advantage, initiates by identifying needs and opportunities and, in general, takes place in the company partner of the EMBRAPII System, which develops projects with the accredited R&D institutions.
EMBRAPII elaborated an operational standard to make research institution a strategic partner that operates where the NIS needs the most: on the intermediate stage. This stage is placed between the
development of the irst proofs of concepts and the
development of products and processes, in the context
of companies able to put the inal product for sale,
with embedded technology. Therefore, EMBRAPII performs a wide range of project types, always in partnership with companies, in order to facilitate introduction of new technologies in products, processes or business models.
Based on prior knowledge and according to studies of authors of the Innovation, Technology and Product
Management ield (Clark & Wheelwright (1993),
Clausing (1993), Cooper (2006), Creveling et al. (2003), Schulz et al. (2000), Sheasley (2000), EMBRAPII sought to address actions for the R&D Institutes’ improvement on the EOE standard, providing them capability to: 1) plan technological strategies; 2) plan compatible business that support its actions; 3) manage portfolio of insights (demand) and its projects; 4) establish business processes which allow operational improvement; 5) develop its skills constantly.
The EOE system is a reference standard that can be used as a tool from which EMBRAPII shares and encourages practices improvement in all Units in order
to obtain an eficient management. The purpose of using
this standard is to guide business processes’ network and establish a common language among actors to
support EMBRAPII system on the qualiication and
monitoring of accredited R&D Institutes.
• EOE actors is a model that describes roles or a
set of responsibilities explicitly identiied on the
Organizational Structure of each EMBRAPII Unit.
• EOE business processes is a model that describes a set of processes or practices that produce
veriiable results (deliveries) and valuable
for the costumers of EMBRAPII Unit. It was selected and developed four main processes: 1) Business Opportunities Development; 2) RD&I Management; 3) Communication; 4) Financial and Administrative Management.
• Key Performance Indicators are the performance indicators used to show how well EMBRAPII Unit goals are being achieved.
Figure 1 represents the general overview of the interaction between EMBRAPII and its Units by means of the components of EOE System (For further information access EMBRAPII website).
This work considers that any organization that drives research institutions, such as EMBRAPII, can develop a standard management system for its innovation network, as EMBRAPII did with its EOE System. However, it is important that such organizations are concerned about the use of these standards and their application, disseminating and using tools to ensure their implementation.
The proposal of R&D Institutes network evaluation model that uses predetermined management systems is presented on section 4.
4 Building an evaluation model of
R&D Institutes network
The work was organized in action-research stages.
On the irst stage, a research team was built, which
one Master’s Degree student and two researchers from University of São Paulo, specialists on Innovation Management and Product Development, composed. The researchers have broad experience as users of Brazilian NIS development programs as well as of cooperative projects development among universities and enterprises.
The irst step after the team building was the
analysis of literature and prior knowledge of the team, which enabled proposing a set of requirements and general purposes of the model: 1) Assure the agent organization’s and its accredited institutions’ growth; 2) Be a guide for continuous improvement of the institutions management and their business processes; 3) Serve as a common language for using standardized management systems among actors of an innovation network composed by research institutions.
Simultaneously, a detailed bibliographic review was made by seeking articles proposing evaluation of innovation networks, performance measurement systems and audit in innovation networks, as mentioned on section 2. In this context stood out the urge to have means contributing for technological development
and for building a business environment beneicial
to innovation in Brazil.
On the second stage, the EOE System was studied
and identiied as a guidance tool for Brazilian R&D
Institutes network and deined as a good example to
develop the study. After all, as mentioned on section 3, it has the required characteristics for being a model for evaluation process. The result analysis of the selection of the most appropriate concepts and tools for evaluation model was performed among pairs of the research team that sought to bring together the most relevant aspects needed for the proposal.
Applying the knowledge acquired, on the third stage was selected and developed four main components to form the evaluation model.
A summary of activities relating to the evaluation model establishment is represented on Chart 2.
The components of the evaluation model are presented and detailed hereinafter from subsections 4.1 to 4.4.
1. Performance Measurement System (PMS). It is point out as a set of initial performance
indicators used for measuring both eficiency and
effectiveness of actions taken in each research institution and on the system as a whole.
2. Internal Evaluation Process. It was thought to be an improvement process established by the research institution, according to its needs and experience, and to ensure the continuous
search for identiication of problems and their
improvement in daily activities.
3. External Evaluation Process. It was developed as a set of follow-up actions of the results of each R&D Institutes and its evolution, run by the agent organization or by Independent Organization at its order. Three objectives have been proposed: 1) evaluate the adoption of institution processes to the proposed standards in management systems,
such as EOE System; 2) evaluate eficiency on
establishing and achieving goals of resources
appliance; 3) evaluate actions eficiency and
its impact on society.
4. Technical group of Management System improvement. A technical group to discuss and introduce development for the management system as a whole, based on experience acquired by the institutions, in order to support the agent organization on continuous improvement of its management system.
Figure 2 shows a general overview of the proposed
evaluation model. It is suggested that the irst three
components are independent principles of the evaluation model, with distinct features. Component 4 is understood as transverse to the others and, aligned with Bin (2008), corroborates to identify priorities and to provide structural conditions for basing actions to be taken by agents of an innovation system.
Figure 2. General overview of R&D Institutes networks evaluation model (own elaboration).
Chart 2. Activities relating to evaluation model establishment.
Stage 1
Building the research team
Analysis of literature and the team’s prior knowledge Identifying needs
Requirements proposal and general purposes of the model
4.1 Performance measurement system
The adoption of a performance measurement system was conceived to be a tool enabling a continuous evaluation of the research institutions performance in relation to achieving the managing organization’s goals. It is proposed that its operation concerns regular update and evaluation of the performance indicators of business processes, predetermined by rules of management system, as the EOE system cited on section 3, for instance.
Aligned to the proposals deined by Neely et al.
(2002) on section 2, it is suggested that the goal of such tool is to guarantee ongoing monitoring so the agent organization can act and give the necessary
support in case of dificulty or delay on the execution
of actions made by the accredited institutions. It also shall assist on transparency of the system, showing society a sum up of projects and progress in each institution in the face of the investment made.
The veriication shall be made by follow-up of
information that creates the indicators, which might allow an analysis of the environment and operational results of accredited institutions. As mentioned on section 2 and according to Kaplan & Norton (1997) instructions, the indicators shall be aligned to the organization’s strategic goals that run on promoting innovation in their countries. As on the Brazilian case of EMBRAPII, the indicators are aligned to the goals in its management contract with the federal government (For further information access EMBRAPII website (EMBRAPII, 2014)).
4.2 Internal evaluation process
Business processes are among the requirements for adaptation to management systems, such as the EOE system. It is important that research institutions testify the existence of structured processes for continuous improvement of its business processes in the case studied, as the processes of EOE System cited in section 3.
The methods and techniques for improvement are up to the institutions. However, there is a set of actions for use of management system standards, as instructed:
Submission of explanatory documents about the institutions internal system and how it is related to the current standard processes;
• Documents of changes made on the Management System;
• Establishment of an speciic internal organizational structure (Committee, group, team and department) responsible for Improvement Program;
• Incorporation of training activities on continuous improvement techniques and tools and practices of predetermined standards.
It is suggested that speciic training activities on
such standards, including processes, indicators and information systems used by the research institution are developed along with the agent organization, considering the following aspects: objectives, target audience, goals, courses and events grid, calendar for implementation, and responsible for training.
4.2.1 Continuous improvement system
It is important that research institutions take into consideration a management system able to ensure continuous search to identify problems and result in improvements. Thus, is suggested a plan development for implementing this system. To be considered as such, a Continuous Improvement System shall consider four aspects.
• Philosophy. Establish an improvement model and mission, vision and principles for continuous improvement on the organization.
• Processes. Determine processes standard for improvement projects, as the traditional DMAIC, PDCA or another one already used by the organization, with tools and actions.
• People. Development of activities that ensure employees’ training to embody the mindset about management philosophy, having direct impact on the organization’s culture.
• Problems solution. Setting the organization for detection, prioritization and solution of problems on processes.
The deinitions of improvement system, optimization
process and continuous improvement of processes
can be seen on standards such as the Brazilian ISO/
NBR 9001:2005 (ABNT, 2005) and on the outline CMMI-DEV, v 1.2 (SEI, 2006). It shall be mandatory to establish an internal continuous improvement system for all research institutions part of an innovation network. The methodology chosen and the type of standards used can be chosen by the R&D Institute. Below are some examples of improvement models that can be used as reference.
• Six-sigma and six-sigma teams. An approach ruled for quality improvement centered in evaluation methods and statistics measurement.
• Kaizen. continuous improvement (incremental) focused on activities that do not add value, that is, waste removal. It relies on Kaizen Events which are short period events with fast results commitments.
• Problem-Solving Method. It is a dynamic process seeking to solve a certain situation. It aims to increase the likelihood of successfully resolve a situation where a problem has arisen. Several concepts are used to ground its practice, being PDCA (Plan, Do, Check, Act) the most known.
• Good Laboratory Practice (GLP). It is a quality system regarding organizational process and conditions in which non-clinical studies related to health and global environment are planned,
performed, monitored, registered, iled and
reported.
4.2.2 Organizational structure for continuous improvement
To ensure implementation of internal improvement system as well as to assist predetermined rules and standards requirement, it is suggested that research institutions show an organizational structure composed
by qualiied professionals with certiications and speciic coaching on the adopted methodology. The speciic deinition is up to each institution.
The minimum requirement is to have a structure
formed by areas, groups and/or individuals who have as functions well deined responsibilities regarding
continuous improvement, establishing (For detailed
explanation see ISO/NBR 9001:2005 (ABNT, 2005),
CMMI-DEV, V 1.2 (SEI, 2006) and Meier & Liker, 2007):
• Improvement teams. Team responsible for planning and execution of activities of process improvement in organizations, according to the process action plan.
• Improvement program. A set of related projects and supportive infrastructure including purposes, methods, activities, plans and measures of success.
• Quality department. Speciic sector of the
organization that holds staff and speciic expertise
to manage the improvements system.
• Manager committees of process or of continuous improvement. Group of people responsible for evaluating, approving or rejecting proposed
changes and for ensuring implementation of approved changes.
The structure can be simpliied or sophisticated
according to research institution’s needs.
4.2.3 Individual performance evaluation program
The performance management is characterized as a subsystem of personnel management; these people in turn, at research institutions, are in possession of knowledge and skills applied in business processes and innovation projects. Thus, the evaluation of individual performance is closely related to operational performance of the agent organization and of innovation network as a whole, including its processes and indicators, contributing to achieve its goals.
Research institutions can have an individual
performance evaluation system simpliied according to
their realities. It is essential to establish an agreement on the purpose of its development within R&D Institutes since these systems can have different aims. Among them, highlights work scheduling, inputs acquisition for training and staff development, team potential
identiication, skills identiication, feedback about
results and recognition through awards.
The performance evaluation process involves planning, monitoring and work evaluation activities, which main goal is to improve people performance, besides improving work quality and life quality within the institution.
Some performance evaluation models are mentioned
below for exempliication purposes.
• Participatory evaluation for objectives. This method, according to its authors, is democratic, participatory, involving and encouraging. It consists of six moments: establishment of agreed objectives, personal commitment to achieve these objectives, negotiation on allocation of means and resources to achieve the objectives, performance, monitoring of results and comparison with established objectives, intensive retroaction and joint and ongoing evaluation.
• 360o or 180o evaluation. Performance evaluation
taken in circular or semicircular way by all elements that keep any sort of interaction with the individual evaluated. The method is based on great participation of all members of the team; individual performance information is gathered from everyone around.
evaluation program, but it is recommended due to
beneits it can provide, in particular for developing
the institutions skills.
4.3 External evaluation process
It is suggested that every research institution, acting on innovation networks and according to predetermined management system standards, participate on external evaluation actions carried out by three mechanisms: information system, technical
auditing and Certiication process.
4.3.1 Information system
The agent organization, as EMBRAPII, shall use
an information system for managing information low,
updating performance indicators of management system and carrying out competitive benchmarking among accredited institutions. This requirement attendance shall occur without making substitutions or any other changes on current information systems in institutions. For this system to work is fundamental to perform the following processes: 1) Implement
information low; 2) Hold a data repository.
4.3.2 Technical auditing
It is a speciic auditing on Standard Management System, or on part of it, deined and performed ad-hoc
by the agency (EMBRAPII in this case), or a third
registered organization, according to speciic needs. The audit can be made through technical visits and/or
analysis of documents particularly required. The visits have explanatory nature, aiming to gather detailed information on management system of institutions, specific projects, evaluation of problems, improvement and action plan. The research institutions shall keep update registers related to processes provided on standard rule for follow-up like this, following guidelines of subsection 4.2.1 Continuous improvement system.
These audits are addressed to be executed by specialists of standard rules, as the case of EOE System, and it can also be performed by expert professionals on technologies related to the projects, depending on the evaluation scope of the audit in question.
4.3.3 Management system certiication
auditing
This routine was conceived as a mechanism for closer monitoring, in case of detection of inconsistencies,
problems or opportunities for improvement on certiied
research institutions. It shall certify its Management System in order to guarantee accordance with the
standards of the agent organizations. Such certiication
would be a mean of ensuring that institutions introduced
the necessary actions to manage relevant aspects of its activities.
Certiication will provide independent demonstration
that the management system of research institution:
a) conforms to speciied requirements, b) is capable
of achieving a coherent goals and c) is implemented effectively, providing value to own agency organization, its customers and stakeholders, as referenced in the literature and presented in section 2.
The agency can hire a third party auditing to
evaluate, grant and maintain the certiication for the
R&D Institutes. It can may use external institutions base on the trust on the principles of audits of this nature, ensuring in this way:
• Integrity. Engagement of professionals acting with impartiality, honesty and diligence.
• Conidentiality. Discretion on the use and protection of received information.
• Independence. Feedback and evaluation of independent auditors, who are exempt of
tendencies and conlict of interest.
• Evidence-based approach. Rational method
to achieve reliable and veriiable conclusions.
• Fair presentation. Truthfully and accurately reporting.
• Responsibility. Implies evaluating suficient objective evidence on which it is possible to
ground a certiication decision.
It is advised to have speciic documentation showing the certiication scope and ensuring that all people, processes and ields that perform projects with the
research institutions are involved.
4.4 Technical group of Management System Improvement
As mentioned on section 2, evaluation of research institutions is a social construction. In this regard, technical groups of improvement are designed to ensure learning and improvement of the standard used by different agents part of the innovation network.
principles, processes, recommended practices and other standard’s contents.
The goal is to transfer the experience gained by R&D Institutes with the management system in order to subsidize the Agency in continuous improvement the Standard. The existence of the group can help the changes are in line with the expected changes to the internal improvement system of institutions.
5 Final considerations
This paper proposed an evaluation model of research institution networks, based on a mix of internal and external evaluation processes, aligned to a Management System Standard. The model aims to support improvement in National Innovation System’s development. Its idealization emerged from previous experience and bibliographic review. It was considered a lack of instruments and practical actions
regarding NIS improvement and the signiicance of innovation networks evaluation and its beneits as
main aspects for building the proposal. Principles of evaluation taken from literature were use as basis to justify the use of EOE System Standard as main evaluation criterion and to develop its main elements.
Knowledge of S&T evaluation approaches, performance measurement systems and auditing in innovation networks were essential to elaborate the research. As mentioned on section 2, it was considered: 1) use of Management System Standards as main criterion for R&D Institutes to perform an evaluation of its business processes and its operation as a whole; 2) use of three evaluation elements to support decision making in institutions, such as: performance measurement system and internal and external evaluation processes; 3) the need to build groups or teams that can learn, perform evaluations and help with the develop of Management System Standard adopted by networks, such as the example of EOE System, used by EMBRAPII.
The use of the model as well as the standardization process, guided by management systems such as EOE System, require institutional support at the R&D Institutes, so that the necessary resources are allocated for this project. This aspect can be considered a limitation to the model success. Continuity of this work shall enable new adjustments until it becomes as good as possible to their purposes. The research team intends to continue monitoring these experiences and hopes that this article may encourage other research groups to investigate this issue, following this case or replicating it in other countries.
The model can be used in similar applications, such as in networks of university laboratories, technological parks or even R&D Institutes singly.
If its credit is veriied, it can still generate the creation of a national certiication, and international then, in third party type. In addition to the beneits cited for
the development of research institutions, the Standard would offer an interesting mechanism to guide companies in building partnerships for technological development by providing them with an assessment of the preparation in terms of quality of services of these institutions.
Given the absence of similar models, the paper presents this new perspective that can open a new line of research in technology management: on evaluation and auditing to management innovation. The model deserves future research effort in the sense that it can complement it with the possibility of creating a research institutions management platform, thus contributing to the adjustment of the innovation network strategies.
References
Andreassi, T., Campanário, M., Stal, E., & Sbragia, R. (2006). Inovação. Como vencer este desaio empresarial. São Paulo: Clio.
Arbix, G., Salerno, M., & De Negri, J. (2005). O impacto da internacionalização com foco na inovação tecnológica sobre as exportações das firmas brasileiras. Dados, 48(2), 395-442. http://dx.doi.org/10.1590/S0011-52582005000200006.
Assad, A., & Souza, R. (2009). Challenger of Innovation in the area of health: learning in the ongoing debate. Cadernos de História da Ciência, 5(2), 9-19.
Associação Brasileira de Normas Técnicas – ABNT. (2002). NBR/ISO: 19011: Diretrizes para auditoria de sistemas de gestão da qualidade e/ou ambiental. Rio de Janeiro: ABNT.
Associação Brasileira de Normas Técnicas – ABNT. (2005). NBR ISO 9000: Sistema de Gestão da Qualidade - Fundamentos e Vocabulário. Rio de Janeiro: ABNT.
Associação Brasileira de Normas Técnicas – ABNT. (2007). NBR/ISO: 17021: Avaliação de conformidade. Requisitos para organismos que fornecem auditorias e certiicação de sistemas de gestão. Rio de Janeiro: ABNT.
Atkinson, G. (2000). Measuring corporate sustainability. Journal of Environmental Planning and Management, 43(2), 235-252. http://dx.doi.org/10.1080/09640560010694.
Bin, A. (2008). Planejamento e gestão da pesquisa e da inovação: conceitos e instrumentos (Tese de doutorado). Instituto de Geociências, Universidade Estadual de Campinas, Campinas.
Bin, A., Salles-Filho, S. L. M., Paulino, S. R., & Rodrigues, G. S. (2003). Impactos ambientais na agricultura: um método de avaliação de programas tecnológicos. Revista Ciências Administrativas, 9(1), 76-83.
Boly, V., Morel, L., Assielou, N. D. G., & Camargo, M. (2014). Evaluating innovative processes in french firms: methodological proposition for firm innovation capacity evaluation. Research Policy, 43(3), 608-622.
http://dx.doi.org/10.1016/j.respol.2013.09.005.
Bourne, M., Neely, A., Platts, K., & Mills, J. (2002). The success and failure of performance measurement initiatives. International Journal of Operations & Production Management, 22(11), 1288-1310. http://
dx.doi.org/10.1108/01443570210450329.
Cardoza Galdamez, E. V. (2007). Proposta de um sistema de medição de desempenho para clusters industriais de pequenas e médias empresas (Tese de doutorado). Escola de Engenharia de São Carlos, Universidade de São
Paulo, São Carlos. http://dx.doi.org/10.11606/T.18.2007.
tde-05032008-124613.
Chandler, A. D. (1962). Strategy and structure: chapters in the history of the American enterprise. Cambridge: The MIT Press.
Chiesa, V., Coughlan, P., & Voss, C. A. (1996). Development of a technical innovation audit. Journal of Product Innovation Management, 13(2), 105-136. http://dx.doi.
org/10.1016/0737-6782(95)00109-3.
Clark, K. B., & Wheelwright, S. C. (1993). Managing new
product and process development: text and cases. New York: The Free Press.
Clausing, D. (1993). Total quality development: a step-by-step guide to world-class concurrent engineering. New York: American Society of Mechanical Engineers.
Cooper, R. G. (2006). Managing technology development projects. Research Technology Management, 49(6), 23-31.
Creveling, J.L., Slutsky, D., & Antis, C., Jr. (2003). Design for six sigma: in technology & product development. New Jersey: Prentice Hall.
Cross, K., & Lynch, R. (1989). Accounting for competitive performance. Journal of Cost Management for the Manufacturing Industry, 3(1), 20-28.
Empresa Brasileira de Pesquisa e Inovação Industrial – EMBRAPII. (2014). Sistema de Excelência Operacional EMBRAPII (EOE). Brasília: EMBRAPII. Recuperado em 10 de novembro de 2014, de www.embrapii.org.br
Furtado, A. T., Terra, B., Passos, C. A. S., & Plonsky, G. A. (2002). Indicadores de C&T para avaliar os Programas de Cooperação entre Universidade e Indústria: Uma Análise do PADCT III. In: Anais do XVII Simpósio de Gestão da Inovação Tecnológica (15 p., CD-ROM). São Paulo: NPGT-USP.
Gerolamo, M. C. (2003). Proposta de sistematização para o processo de gestão de melhoria e mudanças no desempenho (Dissertação de mestrado). Escola de Engenharia de São Carlos, Universidade de São Paulo, São Carlos.
Hammer, M. (2007). The process audit. Canada: Harvard Business Review.
Hansen, P. B., & Oliveira, L. R. (2009). Proposta de modelo para avaliação sistêmica do desempenho competitivo de
arranjos produtivos: o caso do arranjo coureiro-calçadista do Vale dos Sinos. Produto e Produção, 10(3), 61-75.
Jannuzzi, A., Oliveira, T., & Teixeira, C. (2008). Análise da interação de uma Instituição Cientíica e Tecnológica (ICT) com o setor produtivo: o caso do Instituto Nacional de Tecnologia (INT). Rio de Janeiro: INT.
Recuperado em 5 de novembro de 2013, de http:// biblio.int.gov.br/phl82/INT_DOCELE/Analise%20 da%20Intera%C3%A7%C3%A3o.pdf
Kamimura, Q. P., & Cornetta, V. K. (2011). Inovação e propriedade intelectual no setor farmacêutico: desafios governamentais e efeitos na saúde pública. In Anais do XXXI Encontro Nacional de Engenharia de Produção (pp. 1-9). Rio de Janeiro: ABEPRO. Recuperado em
5 de novembro de 2014, de http://www.abepro.org.br/ biblioteca/enegep2011_TN_STO_135_855_18348.pdf
Kaplan, R. S., & Norton, D. P. (1997). A estratégia em ação: balanced scorecard (26 ed.). Rio de Janeiro: Campus.
Lengrand, L. (2006). Smart innovation: a practical guide to evaluating Innovation Programs. Brussels-Luxembourg: ECSC-EC-EAEC. 199 p.
Lenz, G. S. (2007). Medição de desempenho: uma proposta de sistema de avaliação de performance para redes horizontais de cooperação (Dissertação de mestrado). Faculdade de Administração, Contabilidade e Economia, Pontifícia Universidade Católica do Rio Grande do Sul, Porto Alegre.
Levén, P., Holmström, J., & Mathiassen, L. (2014). Managing research and innovation networks: evidence from a government sponsored cross-industry program. Research Policy. 43(1), 156-168. http://dx.doi.org/10.1016/j. respol.2013.08.004.
Liu, J. S., Lu, W. H., & Ho, M. H. C. (2014). National
characteristics: innovation systems from the process efficiency perspective. R & D Management, 45(4),
317-338. http://dx.doi.org/10.1111/radm.12067.
Lundvall, B.-A., Johnson, B., Andersen, E. S., & Dalum, B. (2002). National systems of production, innovation and competence building. Research Policy, 31(2), 213-231. http://dx.doi.org/10.1016/S0048-7333(01)00137-8.
Meier, D., & Liker, J. (2007). O modelo Toyota: manual de aplicação. Porto Alegre: Bookman.
Neely, A., Adams, C., & Kennerley, M. (2002). The performance prism: The scorecard for measuring and managing business success. London: Prentice Hall.
Neely, A., Gregory, M., & Platts, K. (2005). Performance measurement system design: A literature review and research agenda. International Journal of Operations & Production Management, 25, 1228-1263.
Nobelius, D. (2004). Linking product development to applied research: transfer experiences from automotive company. Technovation, 24(04), 321-334. http://dx.doi.
org/10.1016/S0166-4972(02)00073-1.
Schulz, A. P., Clausing, D. P., Fricke, E., & Negele, H. (2000). Development and integration of winning technologies as key to competitive advantage. Systems Engineering, 3(4), 180-211. http://dx.doi.org/10.1002/1520-6858(2000)3:4<180::AID-SYS2>3.0.CO;2-H.
Sheasley, W. D. (2000). Taking an options approach to
new technology development. Research Technology Management, 43(6), 37-43.
Software Engineering Institute – SEI. (2006). Improving process for better products. CMMI-DEV, V1.2. Pittsburg: SEI.
Valmohammadi, C., & Servati, A. (2011). Performance measurement system implementation using Balanced Scorecard and statistical methods. International Journal of Productivity and Performance Management, 60(5), 493-511. http://dx.doi.org/10.1108/17410401111140400.
Viotti, E. B. (2002). National learning systems - a new approach on technological change in late industrializing economies and evidences from the cases of Brazil and South Korea. Technological Forecasting and Social Change, 69(7), 653-680. http://dx.doi.org/10.1016/
S0040-1625(01)00167-6.
Yam, R., Guan, J. C., Pun, K. F., & Tang, E. P. Y. (2004). An audit of technological innovation capabilities in chinese firms: some empirical findings in Beijing, China. Research Policy, 33(8), 1123-1140. http://dx.doi.
org/10.1016/j.respol.2004.05.004.
Zackiewicz, M. (2005). Trajetórias e desaios da avaliação em ciência, tecnologia e inovação (Tese de doutorado). Instituto de Geociências, Universidade Estadual de Campinas, Campinas.
performance measurement. Journal Computers and Industrial Engineering, 60(2), 279-290.
Oliveira, J. F. G., & Telles, L. O. (2011). O papel dos institutos públicos de pesquisa na aceleração do processo de inovação empresarial no Brasil. Revista USP, 89, 204-217.
Organização para a Cooperação e Desenvolvimento Econômico – OCDE. (2005). Manual de Oslo: proposta de diretrizes para coleta e interpretação de dados sobre inovação tecnológica (3 ed.). Brasília: OCDE, FINEP.
Organização para a Cooperação e Desenvolvimento Econômico – OCDE. (2009). New nature of innovation. Copenhagen: OCDE. Recuperado em 6 de fevereiro de
2014, de http://www.oecd.org
Paulino, S. R., Salles-Filho, S., & Bin, A. (2007). Avaliação de programas tecnológicos: impactos ambientais da inovação na agricultura. In R. Casas, C. Fuentes & A. O. Vera-Cruz (Org.), Acumulación de capacidade tecnológicas, aprendizaje y cooperación en la esfera global y local (Vol. 1, pp. 3-26). Ciudad de México: Universidad Autónoma de Metropolitana.
Penrose, E. (2006). A teoria do crescimento da irma ( T. Szmrecsányi, Trad.). Campinas: Editora da Unicamp.
Rutman, L. (Ed.). (1977). Planning an evaluation study. In L. Rutman. Evaluation research methods: a basic guide (pp. 13-38). Beverly Hills: Sage Publications.
Salles-Filho, S., Avila, F. D., Sepulveda, J., & Colugnati, F. A. B. (2010). Multidimensional assessment of technology and innovation programs: the impact evaluation of INCAGRO-Peru. Research Evaluation, 19, 1, 361-372.
