“An Embedded Software Component Quality Verification Framework”
Fernando Ferreira de Carvalho
PHD THESIS PROPOSE
Universidade Federal de Pernambuco [email protected] www.cin.ufpe.br/~posgraduacao
RECIFE, 11/2008
CENTRO DE INFORMÁTICA
PÓS-GRADUAÇÃO EM CIÊNCIA DA COMPUTAÇÃO
Fernando Ferreira de Carvalho
“An Embedded Software Component Quality Verification Framework"
ESTE TRABALHO FOI APRESENTADO À PÓS-GRADUAÇÃO EM CIÊNCIA DA COMPUTAÇÃO DO CENTRO DE INFORMÁTICA DA UNIVERSIDADE FEDERAL DE PERNAMBUCO COMO REQUISITO PARCIAL PARA OBTENÇÃO DO GRAU DE DOUTOR EM CIÊNCIA DA COMPUTAÇÃO.
A PHD. THESIS PRESENTED TO THE FEDERAL UNIVERSITY OF PERNAMBUCO IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF PHD. IN COMPUTER SCIENCE.
ADVISOR: Silvio Lemos Meira
RECIFE, NOVEMBER/2008
R ESUMO
Um dos maiores desafios para indústria hoje é fornecer produtos com alto nível de qualidade e funcionalidade com um baixo custo e curto tempo de desenvolvimento. Os requisitos de custo e tempo de desenvolvimento têm sido abordados com bastante êxito pela engenharia de software baseada em componentes (CBSE). A tecnologia baseada em componentes é amplamente utilizada nas indústrias mecânica e de componentes eletrônicos, entre outras áreas, mas na indústria de software embarcados ainda não alcançou o nível desejado. É de conhecimento geral que a força de um corrente está intimamente ligada com a força de cada elemento que a compõe. Da mesma forma, a qualidade dos atributos de um sistema embarcado, como seu desempenho, sua confiabilidade e segurança, dependem do nível de qualidade de cada componente que integra o sistema. Por este motivo, a abordagem baseada em componentes, embora atraente para muitas razões, é difícil utilizar em domínios nos quais atributos de qualidade são de vital importância. Em CBSE, utilização de mecanismos apropriados de pesquisa, seleção e avaliação do processo de componentes, é considerada o ponto chave para o desenvolvimento de qualquer sistema baseado em componentes. Por este e outros motivos, esta tese propõe um Framework para Verificação da Qualidade de Componentes de Software Embarcados, que prove mecanismos de avaliação dos atributos de qualidade do componente de software embarcado, baseando-se módulos bem definidos que se complementam uns aos outros em busca da qualidade dos componentes embarcados, de forma consistente. O sonho do desenvolvedor de sistemas
utilizados nos sistemas mecânicos. Pela introdução de verificação e certificação da qualidade de componentes de software no projeto de sistemas embarcados, esta tese é um pequeno passo para alcançar este sonho.
A bstract
One of the major challenges to industry today is to provide products with high degrees of quality and functionality at low cost and short time to market. The cost and time to market requirements have quite successfully been addressed by the component-based software engineering (CBSE). Unfortunately, satisfactory solutions for handling quality are not yet available. Component technologies are widely used in the electronics and mechanics industries, but software industry has not reached the desired level. It is well known that the strength of a chain is proportional to the strength of its interconnected elements. Similar to this concept, the quality attributes of the final embedded system, such as its performance, scalability or reliability, depends on the quality level of the components that integrate the system. For these reasons, the component-based approach, although attractive for many reasons, is difficult to apply in domains in which quality attributes are of primary importance. In CBSE, the proper search, selection and evaluation process of components is considered the corner stone for the development of any effective component-based system. For these and other reasons, this thesis proposes an Embedded Software Component Quality Verification Framework that provides mechanisms for assessing quality attributes of embedded software components, based upon well-defined modules that complement each other looking for the embedded component quality in a consistent way. Having the means to reason about the qualities of an embedded software design in the same way as one can reason about the
embedded systems design, this thesis is a small step towards fulfilling this dream.
L ist of Figures
Figure 1.1. The Framework for Software Reuse...18
Figure 1.2. The Component Certification Process was divided into two parts:...19
Figure 1.3. Embedded Software Component Quality Verification Framework...20
Figure 2.1. An embedded system encompasses the CPU as well as many other resources...31
Figure 2.2. Component technology for small embedded Systems....33
Figure 2.3. The results. Y-axis: priority of quality attributes in a scale 1 (highest),...41
Figure 3.1. Structure of Prediction-Enabled Component Technology (Wallnau, 2003)...54
Figure 3.2. SQuaRE Architecture...63
Figure 3.3. ISO/IEC 25040...68
Figure 3.4. Process of obtaining, certifying and storing components ...71
Figure 3.5. Research on embedded software component quality and certification timeline...72
Figure 4.1. Embedded Software Component Quality Verification Framework...78
Figure 4.2. Relations among the quality model elements...90
Figure 4.3.1 A different evaluation level for each embedded component characteristics...100
Figure 4.5.1. Component Certification Process...119
Figure 4.5.2. Establish Evaluation Requirements activity...120
Figure 4.5.3. Specify the Evaluation activity...125
Figure 4.5.4. Design the Evaluation activity...129
Figure 4.5.5. Execute the Evaluation activity...133
L ist of Tables
Table 2.2. Summary of relevant characteristics in Crnkovic’s
research...37
Table 2.2.2. Priority classes used to classify the importance of the different quality attributes...40
Table 3.1. Software quality standards...58
Table 3.2. IEC61131...59
Table 3.3. DO-178B...61
Table 3.4. Characteristics and Sub-Characteristics in SQuaRE project...67
Table 4.1. Changes in the Proposed Embedded software Component Quality...83
Model, in relation to ISO/IEC 25010...83
Table 4.2. The Proposed Component Quality Model, with the sub- characteristics...89
divided into two kinds: runtime and life-cycle...89
Table 4.2.4. Embedded software Component Quality Attributes for.91 Sub-Characteristics those are observable at Runtime and Life-cycle. ...91
Table 4.2.5 Additional Information...98
Table 4.3.1. Guidelines for selecting evaluation level...101
Table 4.3.2. Embedded software component Maturity Model...105
Table 4.3.3. Embedded Quality Attribute X Evaluation Techniques. ...106
Table 4.5.1. Example of Importance’s definition...123
Table 4.5.2. Example of Quality Attributes Definition...125
Table 4.5.3. Example of Define EMM...126
Table 4.5.4. Example of Tools Selection...130
Table 4.5.5. Example of Data Collection...134
Table 5.3. Cronogram...140
A cronyms
Terms - Descriptions
B2B - Business to Business
CBD - Component-Based Development
CBSE - Component-Based Software Engineering
CMU/SEI - Carnegie Mellon University’s Software Engineering Institute
COTS - Commercial Off-The-Self
CBSD - Component-Based Software Development COM - Component Object Model
CCM - CORBA Component Model
CMM - Capability Maturity Model CMMI - Capability Maturity Model Integrated
EQM - Embedded software component Quality Model
EMM - Embedded software component Maturity Model GQM - Goal Question Metric Paradigm
ISO - International Organization for Standardization IEC - International Electro-technical Commission
PECT - Prediction-Enabled Component Technology PACC - Predictable Assembly from Certifiable Components
RiSE - Reuse in Software Engineering group
SPICE - Software Process Improvement and Capability dEtermination
OMG - Object Management Group
XML - eXtensible Markup Language
C ontents
1 Introduction...11
1.1 Motivation...14
1.1.1 The needs and priorities in research...20
1.2 Problem formulation...24
1.3 Proposed solution...25
1.4 Out of Scope...26
1.5 Statement of Contributions...28
1.6 Organization of the Thesis...28
2 Embedded systems design...30
2.1 Basic concepts for Component-Based embedded systems...31
2.1.1 Component-based approach for small embedded systems 32 2.1.2 Component-based approach for large embedded systems 33
2.2 Specific requirements for embedded systems...35
2.2.1 Industrial Automation...37
2.2.2 Automotive...38
2.2.3 Medical...41
2.2.4 Consumer Electronics...42
2.2.5 Other domains...43
2.3 The needs and priorities in research...43
3 Embedded Software Component Quality and Certification: A Survey...46
3.1 Failures in Software Component Certification...56
3.1.1 Failure in National Information Assurance Partnership (NIAP)...57
3.1.2 Failure in IEEE...57
3.2 Standardization: Software Quality and Certification 57
3.2.1 ISO/IEC 61131...593.2.2 RTCA DO 178B...60
3.2.3 ISO/IEC 61508...62
3.3 Software Component Certification...69
3.4 Summary of the Study...72
3.5 Summary...73
4 Embedded software Component Quality Verification Framework...74
4.1 Overview of the Framework...75
4.2 Embedded software component Quality Model...78
4.2.1 The Embedded software component Quality Model (EQM) 80 4.2.2 Changes in relation to ISO/IEC 25010...83
4.2.2.1 Quality sub-characteristics that were created from ISO/IEC 25010 84 4.2.2.2 Quality sub-characteristics that were removed from ISO/IEC 25010 86 4.2.2.3 Quality sub-characteristics that were renamed from ISO/IEC 25010 87 4.2.2.4 Quality characteristics that were extended from ISO/IEC 25010 88 4.2.3 Summary...89
4.2.4 Embedded software Component Quality Attributes...90
4.2.5 Other relevant Component Information...98
4.3 Maturity Level Evaluation Techniques...99
4.3.1 Embedded software component Maturity Model (EMM) 100
4.4 Metrics Approach...109
4.4.1 Metrics to track the EMM Properties...114
4.4.2 Metrics to track the Certification Techniques Properties 115 4.4.3 Metrics to track the Certification Process Properties. .116
4.5 Component Certification Process...117
4.5.1 The Component Certification Process...118
4.5.1.1 Establish Evaluation Requirements activity...119
4.5.1.2 Specify the Evaluation activity...124
4.5.1.3 Design the Evaluation activity...129
4.5.1.4 Execute the Evaluation activity...132
4.5.1.5 Process Summary...135
5 Conclusion and future works...136
5.1 Contributions...137
5.2 Future Work...139
5.3 Chronogram...140
6 Bibliographic References...141
1 Introduction
Embedded system is at the heart of many systems in use today.
Added value to products is to a large extent provided by the software. Furthermore, production cost reduction is imperative and is often achieved by introducing software that permits the use of less complex hardware. Domains in which the use of software is now essential include, among others, the automotive, medical systems, process control, and manufacturing industries. Embedded systems are often systems consisting of software and hardware, the software part incorporates many software components that must cooperate without fail. It is platform-dependent, consequently difficult to port, upgrade, and customize, and offers limited opportunities for reuse, even within a single application domain.
The demands that companies in these electronic products must satisfy include low production costs, short time to market and high quality. The cost and time to market issue is addressed by means of the rapidly emerging Component-based Development (CBD) approach. Adding new functionalities to existing products must be done at low cost and high quality. In CBD, software systems are built as an assembly of components already developed and prepared for integration.
The potential benefits of component-based development are as attractive in the domain of embedded systems as they are in other areas of the software industry. These include reduction of costs, improved quality, specialization of expertise, effective management of complexity, reduced time to market, increased productivity, a greater degree of consistency, and a wider range of usability (Brown, 2000). Additionally, CBD offers an opportunity to increase productivity by providing natural units for reuse (components and architectures), by raising the level of abstraction for system construction, and by separating services from their configurations to facilitate evolution (Müller et al., 2002).
However, these benefits are much more difficult to realize in embedded systems development than in other areas of software engineering because the problems of composing extra-functional requirements such as quality, reliability and performance are much more acute. When building new applications from existing components it is not only necessary to ensure that they behave as expected, but also that these extra-functional properties are composed correctly. Until recently it was difficult to do this in an economically viable way, but recent trends have combined to make component-based software engineering as imperative for embedded systems development as it is in other domains. According to Atkinson (Atkinson at al., 2005), there are three important trends with embedded domain:
The shear growth in the number and ubiquity of embedded systems in the world around us;
The growth in the size and complexity of software in embedded systems.
The creation of larger product families in order to tailor products variants to the needs of specific customers and/or market segments.
New functionality is frequently added to products to meet market requirements. The new functionality is often introduced by adding components to the system. This is the reason why the component-based development approach is attractive to industry.
Different classes of components are used in different industrial settings, and different component technologies target different application domains. Industrial component technologies such as IEC 61131-3 (ISO/IEC 1131-3, 1995) and Koala (Van Ommering, 2002) are used to build applications. Other component classes are used to provide the infrastructure in systems for examples, databases, real- time operating systems, user interfaces and communication components. This diversity means that it is possible to use components at different levels in a system.
The quality aspects of software products are not, however, addressed adequately by component-based development.
Component-based applications are sensitive to evolution of the system. As components and applications have separate lifecycles and different kinds of requirements, there is some risk that a component will not completely satisfy the particular requirements of certain applications or that it may have characteristics unknown to the application developer. When introducing changes on the application level (changes such as updating the operating system, updating other components, changes in the application), there is a risk that the change introduced will reduce the reliability of the system due to different unforeseen mismatches (Larsson, 2000).
Components are in general considered as black boxes with little or no information easily accessible. The information needed from the software components relates to their quality attributes. If the components’ attributes are not known, the attributes of the system of which they are part of is even more difficult to reason about. Industrial products are required to meet high functional and
quality standards. Customers often pay for the functionality they require and take the quality of the product for granted. This business model influences the way software is developed. Unfortunately, as software developers tend to focus on delivering the functionality without giving the quality requirements the same priority it is possible to ensure the quality characteristics and the functionality of a product by means of extensive testing, but this is generally an expensive and inefficient approach. A much more effective approach would be to ensure the quality attributes of the product on the basis of the quality of the components of which it is constructed.
A software program computes functions with certain functional attributes. There are other attributes that we care for more than the functional ones, these attributes are expressed in terms of quality attributes. A quality attribute is originally the characteristic of a system but today we also recognize quality attributes of software.
Security, safety, performance and dependability are examples of system characteristics. Quality attributes are primarily attributes of systems which more and more are achieved by software.
Quality attributes are often referred to as properties, non- functional or extra-functional attributes because they relate to the quality of the component and not explicitly to the component functionality (Barbacci et al, 1995). The quality attributes describe the characteristics of a component. The properties of a component are the concrete accessible values that represent the characteristics.
A component can express its quality attributes in terms of its properties.
1.1 Motivation
This creates a big challenge for embedded-software development. In the years to come, the key to success will be the
ability to successfully develop high-quality embedded systems and software on time. As the complexity, number, and diversity of applications increase, more and more companies are having trouble achieving sufficient product quality and timely delivery. To optimize the timeliness, productivity, and quality of embedded software development, companies must apply software engineering technologies that are appropriate for specific situations.
Unfortunately, the many available software development technologies do not take into account the specific needs of embedded-systems development. This development is fundamentally different from that of non embedded systems. Technologies for the development of embedded systems should address specific constraints such as severe timing constraints, limited memory and power use, predefined hardware platform technology, and hardware costs. Existing development technologies do not address their specific impact on, or necessary customization for, the embedded domain. Nor do these technologies give developers any indication of how to apply them to specific areas in this domain—for example, automotive systems, telecommunications, or consumer electronics.
Consequently, tailoring a technology for a specific use is difficult.
Furthermore, the embedded domain is driven by reliability factors, cost factors, and time to market. So, this embedded domain needs specifically targeted development technologies. In industry, the general feeling is that the current practice of embedded software development is unsatisfactory. However, changes to the development process must be gradual; a direction must be supplied.
Embedded systems comprise a scale from ultra small devices with simple functionality, through small systems with sophisticated functions, to large, possibly distributed systems, where the management of the complexity is the main challenge. Further, it can distinguish between systems produced in large quantities, in which the low production costs are very important and low-volume
products in which the system dependability is the most important feature. All these different requirements have impact on feasibility, on use, and on approach in component-based development.
There is no universal definition of an embedded system.
However, there is a certain consensus that the following features are common to most embedded systems (Camposano & Wilberg, 1996):
They are part of a larger system (host system), hence the term embedded, with which they continuously or frequently interact. Usually, the embedded system serves as a control unit inside the host system.
They have a dedicated functionality and are not intended to be reprogrammable by the end-users. Once an embedded system is built, its functionality does not change throughout its lifetime. For example, a device controlling the engine of a car will probably never be reprogrammed to decode Mp3s. A desktop computer, on the other hand, has a wide range of functionality, including web browsing, word processing, gaming, advanced scientific calculator, etc.
They have real-time behavior. The systems must, in general, respond to their environment in a timely manner.
They consist of both hardware and software components.
In order to cope with the wide and unpredictable range of applications, the hardware of a general purpose computer has to be meticulously designed with the risk of wasting resources.
However, since the set of applications to be run on an embedded system is known at design-time, including their performance requirements, the hardware can be tuned at design-time for best performance at minimal cost. Similarly, software must also be optimized to build a globally efficient HW/SW system.
There are many domains in which embedded systems are used extensively. Some of them are: Telecommunication, avionics and aerospace, transportation, computer games, home electronics, navigation systems, etc. While there are many similarities between these domains there are also very different requirements for their functional and extra-functional properties. The consequences are that the requirements for component-based technologies are different, and consequently it expects to have one component model that summarizes all quality attributes in common in different domains. The expectations are that only one component models will exist, some common characteristics, such as basic principles of component specifications through interfaces, basic composition and run-time services, certain patterns, and similar.
All these characteristics have strong implications on requirements. Most of the requirements of embedded systems are related to non-functional characteristics, generally designated as extra-functional properties or quality attributes (Crnkovic, 2005).
These properties can be classified in run-time and life cycle extra- functional properties.
A common characteristic of all systems is the increasing importance of software. For example, software development costs for industrial robots today constitute about 75% of total costs, while in car industry it is currently about 30%. Some ten to fifteen years ago this number was about 25% for robots and negligible for cars (Crnkovic, 2005). A second common characteristic is increasing interoperability. While previously the embedded systems were mainly used for controlling different processes today they are integrated with information systems of infotainment technologies
One of the most compelling reasons for adopting component- based approaches in embedded software design is the premise of reuse. The idea is to build software from existing components
primarily by assembling and replacing interoperable parts. The implications for reduced development time and improved product quality make this approach very attractive (Krueger, 1992).
Software reuse is not a new idea. Since McElroy’s pioneer work, “Mass Produced Software Components” (McElroy, 1968), the idea of reusing software components on a large scale is being pursued by developers and research groups. This effort is reflected in the literature, which is very rich in this particular area of software engineering.
McIlroy’s work does not consider an essential requirement for these systems: the assets certification. In a real environment, a developer that retrieves a faulty component from the repository would certainly lose his trust in the system, becoming discouraged to make new queries. Thus, it is extremely important to assert the quality of the assets that are stored into the repository before making them available for reuse. Despite this importance, the software engineering community had not explored these issues until recently. In this way, a new research area arose: components certification and quality assurance (Wohlin et al., 1994), (Mingins et al., 1998), (Morris et al., 2001), (Schmidt, 2003), (Wallnau, 2003).
However, several questions still remain unanswered, such as:
(i) How certification should be carried out?
(ii) What are the requirements for a certification process?
and,
(iii) Who should perform it? (Goulão et al., 2002a).
This is the reason why there is still no well-defined standard to perform component certification (Voas et al., 2000), (Morris et al., 2001).
This thesis addresses embedded software component quality issues with a particular emphasis on component quality certification
for reuse proposes. It also covers issues related to design of component-based embedded systems. This thesis is investigating effective ways to demonstrate that embedded component certification is not only possible and practically viable, but also directly applicable in the embedded software design. Through certification, some benefits can be achieved, such as: higher quality levels, reduced maintenance time, investment return, reduced time- to-market, among others. According to Weber et al. (Weber et al., 2002), the need for quality assurance in software development has exponentially increased in the past few years.
Moreover, this thesis is a part of a big project that aims to develop a robust framework for software reuse (Almeida et al., 2004), in order to establish a standard for component development;
to develop a repository system; and to develop a general software component certification process. This project has been developed in collaboration between the industry and academia (the RiSE group1 and three other universities), in order to generate a well-defined model for developing, evaluating quality, storing and, subsequently, making it possible for software factories to reuse software components.
The framework (Figure 1.1) that is being developed has two layers. The first layer (on the left) is composed of best practices related to software reuse. Non-technical factors, such as education, training, incentives, and organizational management are considered.
This layer constitutes a fundamental step prior to the introduction of the framework in the organizations. The second layer (on the right) is composed of important technical aspects related to software reuse, such as processes, environments, tools, and a certification process, which is the focus of this thesis.
Figure 1.1. The Framework for Software Reuse.
However, to better support the embedded domain and our specific quality characteristics, the component certification process was divided into two parts. The first of these, focuses on general propose software component certification, in general, the software component used in IBM-PC compatible. The other, is a component certification processes that aligns to specific requirements and constraints to develop quality characteristics for embedded domain.
Figure 1.2. The Component Certification Process was divided into two parts:
i – General Propose Certification and ii - Embedded Component Certification.
The process of certifying components proposed is not a simple one. First, there should be an Embedded software component Quality Model (EQM). Differently from other software product quality models, such as (McCall et al., 1977), (Boehm et al., 1978), (Hyatt et al., 1996), (ISO/IEC 9126, 2001), (Georgiadou, 2003), this model should consider Component-Based Development (CBD) characteristics applied to embedded domain, and describe attributes
that are specific to the promotion of reuse. With the availability of an embedded software component quality model, there must be a series of techniques that allow one to evaluate whether a component conforms to the model in different level of maturity. The correct usage of these techniques should follow a well-defined and controllable certification process. Finally, a set of metrics is required, in order to track the components properties and the enactment of the certification process. These four main issues:
(i) Embedded software component Quality Model, (ii) Maturity Level Evaluation Techniques,
(iii) Metrics Approach, and (iv) Certification Process,
Are the modules of an Embedded Software Component Quality Verification Framework (Figure 1.3) that is being investigated as a part of the RiSE project.
Figure 1.3. Embedded Software Component Quality Verification Framework.
The framework will allow that the components produced in a Software Reuse Environment are certified before being stored on a Repository System. In this way, software engineers would have a greater degree of trust in the components that are being reused.
1.1.1
The needs and priorities in researchThe component-based approach on system level, where hardware components are designed with embedded software, has been successfully used for many years. Also large-grain generic components like protocol stacks, RTOSs, etc. have been used for a long time. In addition to this, technology supporting a component- based approach has been developed; either in the form of proprietary component models, or by using reduced versions of some widely used component models.
Still, there are needs for a number of improvements (Crnkovic, 2005). Some of them are the following (differently important for different domains):
There is a lack of widely adopted component technology standards which are suitable for embedded systems. For smaller-size embedded systems, it is important that a system composed of components can be optimized for speed and memory consumption, which is still missing in most of the component technologies available today.
There is also a need for interoperability between different component technologies. Specification technology is not sufficiently developed to guarantee a priori component interoperability.
Component certification: In order to transfer components across organizations, techniques and procedures should be developed for ensuring the trustworthiness of components.
Most current component technologies do not support important extra-functional properties.
There is a need for generic platform services, for, e.g., security and availability.
Tools that support component based development are still lacking.
There is a lack of efficient implementations of component frameworks (i.e., middleware), which have low requirements
on memory and processing power. Major needs for the further development of component technology for embedded systems are the following (Brinksma et al., 2001).
Need for adopted component models and frameworks for embedded systems. A problem is that many application domains have application-dependent requirements on such a technology.
Need for light-weight implementations of component frameworks. In order to support more advanced features in component-based systems, the run-time platform must provide certain services, which however must use only limited resources.
Obtaining extra-functional properties of components: Timing and performance properties are usually obtained from components by measurement, usually by means of simulation.
Problems with this approach are that the results depend crucially on the environment (model) used for the measurements may not be valid in other environments, and that the results may depend on factors which cannot easily be controlled. Techniques should be developed for overcoming these problems, thereby obtaining more reliable specifications of component properties.
Platform and vendor independence: Many current component technologies are rather tightly bound to a particular platform (either run-time platform or design platform). This means that components only make sense in the context of a particular platform.
Efforts to predict system properties: The analysis of many global properties from component properties is hindered by inherent complexity issues. Efforts should be directed to finding techniques for coping with this complexity.
Component noninterference: Particularly in safety-critical applications, there is a need to ensure separation and
protection between component implementations, in terms of memory protection, resource usage, etc.
Tool support: The adoption of component technology depends on the development of tool support. The clearly identified priorities of CBSE for embedded systems are:
Predicting system properties. A research challenge today is to predict system properties from the component properties. This is interesting for system integration, to achieve predictability, etc.
Development of widely adopted component models for real- time systems. Such a model should be supported by technology for generating necessary runtime infrastructure (which must be light-weight), generation of monitors to check conformance with contracts, etc.
Further, according to Voas (Voas, 1998), to foster an emerging software component marketplace, it must be clear for buyers whether a component’s impact is positive or negative. Ideally, buyers should have this information before buying a component. Component buyers could then choose an appropriate component according to its certification level. With this information, system builders could make better design decisions and be less fearful of liability concerns, and component vendors could expect a growing marketplace for their products.
The Carnegie Mellon University’s Software Engineering Institute (CMU/SEI) (Bass et al., 2000) studied industry trends in the use of software components from technical and business perspectives. A distinct set of inhibitors to adopting software component technology emerged from the conducted surveys and interviews of earlier adopters of software component technology.
From this data and from the interviews, the study concludes that the market perceives the following key inhibitors for component adoption, presented here in decreasing order of importance:
Lack of available components;
Lack of stable standards for component technology;
Lack of certified components; and
Lack of an engineering method to consistently produce quality systems from components.
The software engineering community is already attempting to reduce the gaps related to the two first inhibitors.
However, in relation to the third inhibitor, the community is still a fledgling. Further research is required in order to assure the production of certified components, especially when combined with the lack of component-based software engineering techniques that deliver predictable properties (the last inhibitor).
Important researches on software components, such as Heineman (Heineman et al., 2001), Councill in (Councill, 2001), Crnkovic (Crnkovic, 2005), Wallnau (Wallnau, 2003), Wallnau (Wallnau, 2004), Schneider & Han (Schneider & Han, 2004) and Andreou & Tziakouris (Andreou & Tziakouris, 2007) indicates that the future of software components is certification. These authors state that certification is a necessary precondition for CBSE to be successfully adopted and to achieve the associated economic and social benefits that CBSE could yield. With the success of CBSE, software developers will have more time to develop, instead of spending their time addressing problems associated with understanding and fixing someone else’s code. Certified components used during development will have predetermined and well established criteria, thus reducing the risk of system failure and
increasing the likelihood that the system will comply with design standards.
1.2 Problem formulation
The previous section stated that designers increasingly often build systems using reusable components due to the complexity of their designs. Therefore, there is an increasing need to efficiently and effectively qualify such systems. Quality assurance, in particular certification, which can effectively cope with this situation and take advantage of the component-based structure, need to be developed.
The evaluation of the quality of components (i.e. the assessment of their quality attributes) needs to be done by independent parties, at least until software vendors acquire the level of maturity that hardware vendors currently have. The software industry still far from counting with the hardware data sheets and catalogues available for hardware components that describe all their characteristics. However, it is necessary to have them for software components too if it wants to talk about a “real” Component-based Software Engineering. Many organizations struggle in their attempts to select and evaluate an appropriate component in their system. For this reason, a well-defined and consistent software component quality assurance is essential for the component market adoption (i.e. without an efficient mechanism to select/evaluate the component quality.
According to Morris et al. (Morris et al., 2001), there is a lack of an effective assessment of software components in general.
Besides, the international standards that address the software products’ quality issues (in particular, those from ISO and IEEE) have shown to be too general for dealing with the specific characteristics of embedded components. While some of their characteristics are appropriate to the evaluation of components in embedded domain, others are not well suited for that task.
However, the main drawback of the existing international standards, in this case the ISO/IEC 25010, is that they provide very generic quality models and guidelines, which are very difficult to apply to specific domains such as embedded components and CBSE.
Thus, the quality characteristics of this model should be analyzed in order to define and adequate to the component quality characteristics context.
A unified and prioritized set of CBSE requirements for trustworthy components is a challenge in itself (Schmidt, 2003). This fact may be due also to the relatively novelty of this area (Goulão et al., 2002a). Moreover, there is a lack of processes, methods, techniques and tools that support the component certification activity (Alvaro et al., 2005). The existent processes and/or methods dealing only with specific aspects of software component (Alvaro et al., 2005) were not evaluated into industrial scenarios. In fact, they are based on the researchers’ experience, and the real efficiency of evaluating software components using these process/methods remains unknown.
One of the keys to success in industry in the future will be the ability to develop high-quality embedded systems and their software on time in order cost reduced to remain competitive. Organizations whose aim is to construct software by integrating components – rather than developing software from scratch – will not be able to meet their objectives if they cannot find sufficient number of components and component versions that satisfy certain functional and quality requirements. Without a quality level, the component usage may have catastrophic results (Jezequel et al., 1997). However, the common belief is that the market components are not reliable and this prevents the emergence of a mature software component market (Trass et al., 2000). Thus, the components market quality problems must be resolved in order to increase the reliability, and
third-party certification programs would help to acquire trust in the market components (Heineman et al., 2001).
1.3 Proposed solution
This work defines an Embedded Software Component Quality Verification Framework, which includes the steps of definition of embedded software component quality, maturity level evaluation techniques, metrics and component certification, based on a set of activities, metrics and guidelines. Moreover, the process is based on the state-of-the-art in the area and its foundations and elements are discussed in details
In this way, the main goals of this thesis were to define Embedded Software Component Quality Verification Framework that is composed of four inter-related modules in order to assure the component quality degree. This framework was proposed with basis in the standards ISO/IEC 25010, ISO/IEC 9126, ISO/IEC 14598 adapted for component context and embedded domain. Different from other approaches in the literature (Goulão et al., 2002b), (Beus- Dukic et al., 2003), (Cho et al., 2001), (Gui and Scott, 2007) that provide only isolated aspects to assure the component quality, this thesis tries to investigate and make available a complete framework with all necessary mechanisms to execute the component evaluation activities.
Through these evaluations it is expected a continuous evolution of the whole framework in a way that the software industry can start to trust in it and evaluate its own software components.
In order to achieve the main objective, the process is based on the following foundations:
First, there should be an embedded software component quality model. With an embedded software component quality
model available, there must be a series of techniques that allow one to evaluate whether a component conforms to the model in different levels of maturity. The correct usage of these techniques should follow a well-defined and controllable certification process. Finally, a set of metrics are needed, in order to track the components properties and the enactment of the certification process. These four main modules:
Embedded software component Quality Model,
Maturity Level Evaluation Techniques,
Metrics Approach, and
Certification Process.
1.4 Out of Scope
As the proposed reuse process is part of a broader context, a set of related aspects will be left out of its scope. Nevertheless, as these aspects were envisioned since the initial definitions of the process, they can be added in the future with some adjustments.
Thus, the following issues are not directly addressed by this work:
Cost Model: Cost estimation is a key requirement for CBSD before the actual development activities can proceed. Cost is a function of the enterprise itself, its particular development process, the selected solution, and the management and availability of the resource during the development project (Cechich et al., 2003), (Mahmooda et al, 2005). A cost model is useful to help the software engineer during the analysis of the software component available to purchase (or to select or to evaluate). However, it makes sense when, first, you have defined processes, methods, techniques and tools to execute the selection and/or the evaluation task in order to identify the cost/benefit to purchase or to evaluate a component.
Formal Proof: Meyer (Meyer, 2003) and Karlsson (Karlsson, 2006) proposes a formal approach in order to acquire trust in
software components. His idea is to build or to use software components with fully proved properties or characteristics. The intention is to develop software components that could be reliable to the software market.
This thesis does not consider cost model because the whole framework that is the basis for embedded software component evaluation will be considered in this first moment and, after that a cost model to help organizations to define if it is viable evaluate certain kinds of components (or not) will be useful. The formal quality assurance is not considered mainly because the software component market is not inclined to formally specify their software components. This kind of approach is highly expensive, in terms of development time and level of expertise that is needed, and component developers still do not know if it is cost effective to spend effort in this direction without having specific requirements such as strict time constraints or high reliability. However, the Embedded software component Maturity Model (EMM) provides formal proof evaluation techniques that could be useful in some scenarios, depending of the customer’s necessities and the cost/benefit to do so;
1.5 Statement of Contributions
As a result of the work presented in this thesis, the following contributions can be enumerated:
An extensive study of the key developments in the field of quality and certification of embedded software component, in an attempt to analyze this research area and identify trends to follow;
A survey of the state-of-the-art of quality and certification of general propose software component in order to understand
and identify the weak and strong points of existing processes;
A development of an Embedded Software Component Quality Verification Framework aiming to provide a complementary mechanism to standards ISO/IEC to evaluate the component quality;
Definition of the Embedded software component Quality Model (EQM), based on the new standard, the Software Product Quality Requirements and Evaluation (SQuaRE) project (ISO/IEC 25000, 2005) and Component Quality Model (CQM) (Alvaro et al., 2006);
A development of the Embedded software component Maturity Model (EMM) in order to provide different thoroughness levels of evaluation techniques and a set of guidelines for selecting those evaluation levels;
A development of the Component Certification Process in order to provide a high quality and consistent evaluation process; and
A development of the Metrics Approach that is composed of a set of valuable measures to be considered as starting point during the component evaluations;
1.6 Organization of the Thesis
The remainder of this thesis is organized as follows.
Chapter 2 presents a brief overview of embedded system design, component-based development areas and requirements for embedded in different domain. The main concepts of these topics are considered.
Chapter 3 presents, in the first part, the survey of the state-of- the-art of the embedded software component quality and certification area that was performed. The failure cases that can be found in the literature are also described in this chapter.
Subsequently, in the final part, this chapter describes the aspects related to the software quality and certification concepts and standardization. The intention is to show that software reuse depends on quality.
Chapter 4 presents the Embedded Software Component Quality Verification Framework proposed and its related modules.
Session 4.2 describes the proposed Embedded software component Quality Model (EQM), showing its characteristics, its sub- characteristics, the quality attributes and the metrics that are related to the model. Session 4.3 presents the Embedded software component Maturity Model (EMM) that is composed of a set of evaluation levels in order to provide flexibility to the component evaluation. Further, a set of guidance is delineated to direct the evaluation team during the selection of levels. Session 4.4 presents the Metrics Approach and the paradigm adopted to define the metrics is also presented. Some few examples of metrics usage are presented. Session 4.5 presents the Software Component Certification Process, describing all activities and steps that should be followed to execute the component evaluation activity in a more controllable way.
Chapter 5 summarizes the main contributions of this work, presents the related works, the concluding remarks and the future work that will be accomplished during next months.
2 Embedded systems design
Embedded systems comprise a scale from ultra small devices with simple functionality, through small systems with sophisticated functions, to large, possibly distributed systems, where the management of the complexity is the main challenge. An embedded system may be represented by a dedicated processor surrounded by dedicated hardware systems, performing very specific tasks (Carvalho & Junior et al., 2004). Further, it can distinguish between systems produced in large quantities, in which the low production costs are very important and low-volume products in which the system dependability is the most important feature. All these different requirements have an impact on feasibility, on use, and on approach in component-based development. A common characteristic of all systems is increasing importance of software.
For example, software development costs for industrial robots currently constitute about 75% of total costs, while in car industry it is currently about 30%. Some ten, fifteen years ago this number was about 25% for robots and negligible for cars (Crnkovic, 2005). A second common characteristic is increasing interoperability. While previously the embedded systems were mainly used for controlling different processes today they are integrated with information systems of infotainment technologies. In this chapter, a short overview of embedded systems design will be shown.
Figure 2.1. An embedded system encompasses the CPU as well as many other resources.
1.7 Basic concepts for Component-Based embedded systems
In classic engineering disciplines a component is a self contained part or subsystem that can be used as a building block in the design of a system. In software engineering, there are many different suggestions for precise definitions of components in component based software development. The best accepted understanding of component in the software industry world is based on Szyperski’s definition (Szyperski et al. 1998). From this definition it can be assumed that a component is an executable unit, and that deployment and composition can be performed at run-time.
In the domains of embedded systems this definition is largely followed, in particular the separation between component implementation and component interface. However the demands on the binary or executable from is not directly followed. A component can be delivered in a form of a source code written in a high-level language, and allows build-time (or design-time) composition. This more liberal view is partly motivated by the embedded systems context, as will be discussed below.
Many important properties of components in embedded systems, such as timing and performance, depend on the characteristics of the underlying hardware platform. Kopetz and Suri (Kopetz and Suri et al., 2003) propose to distinguish between software components and system components. Extra-functional properties, such as performance, cannot be specified for a software component in isolation. Such properties must either be specified with respect to a given hardware platform, or be parameterized on (characteristics of) the underlying platform. A system component, on the other hand, is defined as a self-contained hardware and software subsystem, and can satisfy both functional and extra functional properties.
1.7.1
Component-based approach for small embedded systemsThe specific characteristics of embedded systems lead to specific requirements of component technologies. In particular the approaches in development process and component specifications using interfaces are different form those implemented in the component technologies widely used in other domains.
The component interface summarizes the properties of the component that are externally visible to the other parts of the system. As for embedded systems non-functional properties are as important as functional there is a tendency to include specification of extra-functional properties in the component interface (for example timing properties). This allows more system properties to be determined when the system is designed, i.e. such interface enables verification of system requirements and prediction of system properties from properties of components.
In general-purpose component technologies, the interfaces are usually implemented as object interfaces supporting polymorphism by late binding. While late binding allows connecting of components
that are completely unaware of each other beside the connecting interface, this flexibility comes with a performance penalty and increased risk for system failure. Therefore the dynamic component deployment is not feasible for small embedded systems.
Taking into account all the constraints for real-time and embedded systems, we can conclude that there are several reasons to perform component deployment and composition at design time rather than run-time (Crnkovic & Larsson et al., 2002): This allows composition tools to generate a monolithic firmware for the device from the component-based design and in this way achieve better performance and better predictability of the system behavior. This also enables global optimizations: e.g., in a static component composition known at design time, connections between components could be translated into direct function calls instead of using dynamic event notifications. Finally, verification and prediction of system requirements can be done statically from the given component properties.
This implies that the component-based characteristic is mostly visible at design time. To achieve an efficient development process tools should exist which will provide support for component composition, component adaptation and static verification and prediction of system requirements and properties from the given component properties.
There may also be a need for a run-time environment, which supports the component framework by a set of services. The framework enables component intercommunication (those aspects which are not performed at design time), and (where relevant) control of the behavior of the components. Figure 2.1 shows different environments in a component life cycle. The figure is adopted from (Crnkovic & Larsson et al., 2002).
Figure 2.2. Component technology for small embedded Systems
1.7.2
Component-based approach for large embedded systems For large embedded systems the resource constraints are not the primary concerns. Complexity and interoperability play a much more important role. Also due to complexity, the development of such systems is very expensive and cutting the development costs is highly prioritized. For this reason general-purpose component technologies are of more interest than in the case for small systems.The technology mostly used in large systems is Microsoft COM and recently .NET, and to a smaller extent different implementations of CORBA, although none of these technologies provides support for real-time. The systems using these technologies belong to the category of soft-real time systems. Often a component technology is used as a basis for additional abstraction level support, which is specified either as standards or proprietary solutions. The main reason for wide use of component-based technology is the possibility of reusing solutions in different ranges of products, efficient development tools, standardized specifications and interoperation, and integration between different products.
One successful example of the adoption of a component-based technology is the initiative OPC Foundation (OLE process control Foundation, www.opcfoundation.org), an organization that consists of more than 300 member companies worldwide, it is responsible for a specification that defines a set of standard interfaces based upon Microsoft’s OLE/COM and recently .NET technology. OPC consists of a standard set of interfaces, properties, and methods for use in
process-control and manufacturing-automation applications. OPC provides a common interface for communicating with diverse process-control devices, regardless of the controlling software or devices in the process. The application of the OPC standard interface makes possible interoperability between automation/control applications, field systems/devices and business/office applications.
Another example of a component-based approach is development and use of the standard IEC 61131 (ISO/IEC 61131-3, 1995). IEC 61131 defines a family of languages that includes instruction lists, assembly languages, structured text, a high level language similar to Pascal, ladder diagrams, or function block diagrams (FBD). Function blocks can be viewed as components and interfaces between blocks are released by connecting in-ports and out-ports. Function lock execution may be periodic or event-driven.
IEC 61131 has been successfully used in development of industrial process automation systems for many years.
Large embedded systems that must fulfill real-time requirements usually do not use general-purpose component-based technologies. However in some cases, such as for ABB controllers, a reduced version of COM has been used on top of a real-time operating system (Lüders et al., 2002). The reused version includes facilities for component specification using the interface description language (IDL), and some basic services at run-time such as component deployment has been used. These services have been implemented internally. Different communication protocols and I/O drivers have been identified as components.
1.8 Specific requirements for embedded systems
Embedded systems vary from very small systems to very large systems. For small systems there are strong constrains related to different recourses such as power or memory consumption and others. In most of the cases, embedded systems are real-time
systems. For these as well as for large embedded systems the demands on reliability, functionality, efficiency and other characteristics that depends on domain or application. Finally, in many domains, the product life cycle is very long – in can stretch to several decades.
All these characteristics have strong implications on requirements. Most of the requirements of embedded systems are related to non-functional characteristics, generally designated as extra-functional properties or quality attributes. Unfortunately, the priority quality attributes vary according to domain application.
These properties can be classified in run-time and life cycle extra- functional properties. In this way, research developed by Crnkovic (Crnkovic, 2005) lists four main characteristics that must be considered to embedded systems, they are listed below:
(i) Real-time properties: a violation of time requirements even of a proper functional response violates the system functionality. The real-time properties:
a - Response time or latency, b - Execution time,
c - Worst case execution time, d - Deadline.
(ii) Dependability is defined as an ability of a system to deliver a service that can justifiably be trusted and an ability of a system to avoid failures that are more severe and frequent than is acceptable to the users (Crnkovic, 2005). The main means to attain dependability are related to avoidance of faults: fault prevention, fault tolerance, fault removal and fault forecasting and it is characterized by several attributes (Avižienis et al., 2001):
a - Reliability, b - Availability, c- Integrity,
d- Safety,
e -Confidentiality f - Maintainability.
(iii) Resource consumption. Many embedded systems have strong requirements for low and controlled consumption of different resources. The reasons may be the size of the systems and/or the demands on lower production costs. Examples of such restrictions and constraints are:
a - Power consumption, b - Memory consumption, c - Execution (CPU) time, d - Computation (CPU) power.
(iv) Life cycle properties. In many domains the embedded systems have very long life time running round the clock year after year. During the lifetime of a system several generations of hardware and software technologies can be used. The long life systems must be able to cope with these changes introduced either into the surrounding environment or into the systems themselves.
In this way, the research concludes that many of the most important requirements of the embedded systems are related to extra-functional properties. This implies that development and maintenance of such systems are very costly. In particular activities related to verification and guaranteed behavior (formal verification, modeling, tests, etc.) and maintenance (adaptive maintenance, debugging, regressive testing, etc.) require a lot of effort. For these reasons the technologies and processes that lead to lower costs for these activities are very attractive and desirable.
Table 2.2. Summary of relevant characteristics in Crnkovic’s research.
Characteristics Sub-characteristics Real-time properties Response time or
latency
execution time
worst case execution time
deadline Dependability Reliability
Availability integrity confidentiality safety
Resource consumption
Power consumption computation (CPU) power
memory Consumption execution (CPU) time, Life cycle properties maintainability
1.8.1
Industrial AutomationTypical application domains of industrial automation are in control of industrial processes, power supply, and industrial robots.
Industrial automation domain comprises a large area of control, monitoring and optimization systems. They typically include large pieces of software that have been developed over many years (often several decades). Most control systems are manufactured in rather large volumes, and must to a large extent be configurable to suit a variety of customer contexts. They can be classified according to different levels of control (Crnkovic & M. Larsson, 2002):
(i) Process level (for example, a valve in a water pipeline, a boiler, etc.),
(ii) Field level that concerns sensors, actuators, drivers, etc.
(iii) Group control level that concerns controller devices and applications which control a group of related process level devices in a closed-loop fashion,
(iv) Process control level i.e. operator stations and processing systems with their applications for plant-wide remote supervision and control and overview the entire process to be controlled,
(v) Production or manufacturing management level that includes systems and applications for production planning.
Notice that, even if the higher levels are not embedded, they are of uttermost importance as they need to be interoperable with the lower level which greatly influences the possible choices of the component model and in fine the design choices. The integration requirements have in many cases led to a decision to use component technologies which are not appropriate for embedded systems but provide better integration possibilities. Depending on the level, the nature of the requirements and the implementation will be quite different. In general, the lower the level, the stronger are the real- time requirements (including timing predictability) and the resource limitations. Also, the component based approach will include different concepts at different levels. The most important quality attributes in industrial automation, following the researchers, is:
(i) lowest levels:
a - Availability, b - Timeliness, c - Reliability (ii) higher levels:
a - Performance, b - Usability, and c - Integrability.
1.8.2
AutomotiveTo provide a domain specific classification of the importance of quality attributes for software in vehicles, and discusses how the attributes could be facilitated by a component technology. Åkerholm (Åkerholm et al., 2005) executed a research in main vehicle industries. The research method is divided into three ordered steps:
