Specify the Evaluation activity

quality characteristics presented on the EQM, it is possible that quality characteristics that are not covered on that model are present. In this case, the evaluation team should define the

“new” characteristic and reference it based on considerable works in the literature in order to clarify the definition and the importance of the quality characteristic. Further, the evaluation team should define the sub-characteristics and quality attributes of this “new” characteristic defined and attached all information in the Table 4.5.1 presented above, and, at the end of the component evaluation process the team should analyze if it is interesting to put those external quality characteristics in the EQM;

 Develop the Evaluation Requirements Specification Document: The last step of this activity is the elaboration of a document with all the information collected during the Establish Evaluation Requirements activity and generate the Evaluation Requirements Specification Document which is the main input of the next activity.

The evaluation team should define metrics to track the properties of the quality characteristics and the techniques adopted as well as the whole certification process. An important aspect here is the consideration of the complexity to obtain the data of each metric required and whether the selected metric can represent completely the information required by the evaluation team.

The inputs of this activity are the Evaluation Requirements Specification, the Embedded Component Quality Model (EQM), the Maturity Level Evaluation Techniques and Metrics Approach. The Quality Characteristics Importance, defined in the last activity, acts as control in this activity. Thus, using those assets, the evaluation team will develop the Evaluation Requirements Specification and generate the Specification of the Evaluation as output, according show in Figure 4.5.3.

Figure 4.5.3. Specify the Evaluation activity.

Figure 4.5.3 has shown the steps that should be following to complete executing the Specify the Evaluation activity and will be presented next.

 Specify the Quality Attributes: Based on the characteristics and sub-characteristics defined by the previous activity (developed on Define the Quality Characteristics step

evaluation team needs to define the quality attributes of each sub-characteristics. For those EQM’s quality characteristics, the evaluation team can use the EQM in order to help them in definition of the quality attributes. And, for those that are external characteristics the evaluation team should define the quality attributes to compose and complement the quality characteristic (if this was not performed yet).

Thus, the evaluation team assures the complete definitions of the quality characteristics necessary to evaluate the software components, as shows on Table 4.5.2. After this step, no more characteristics will be added to the evaluation process and, thus, the evaluation team must guarantee the complete definition of those quality characteristics, as much as possible.

Table 4.5.2. Example of Quality Attributes Definition.

Characteristic s

Sub-Characteristics

Quality Attributes

Importance

Functionality Accuracy Correctness 4

Functionality Security Data Encryption 3

… … … …

 Define the Embedded software component Maturity Model (EMM) level: After specify all quality characteristics and attributes, the evaluation team must define which evaluation technique should be useful to evaluate those attributes. The evaluation team should consider the importance of each quality characteristic (developed on Define the Quality Characteristic step), described in the last activity. Thus, they will define the ECMM level that the software component should attend, using a set of guidelines for selecting evaluation level as basis for this decision (presented in session 4.3 on Table 4.3.1). Thus, the evaluation team selects a mix of levels, where each quality characteristic will be evaluated in a certain EMM level (e.g. functionality will be evaluated on EMM I level,

reliability will be evaluated on EMM III level and so on). The report will contain the level achieved for each characteristic separately (e.g. Functionality – EMM II, Reliability – EMM I, Usability – EMM III, etc). The decision of the directions to be followed during the certification should be communicated to the customer until going to the next step. Table 4.5.3 shows the decision of the evaluation techniques for each quality attributes.

Table 4.5.3. Example of Define EMM.

Characteristic s

Sub-Characteristic

s

Quality Attributes

Importance EMM Level/

Evaluation Technique

Functionality Accuracy Correctness 4 II. Black-Box

Testing

Functionality Security Data

Encryption

3 III. Code

Inspection

… … … … …

Further, if any external quality characteristic was proposed in the last activity (developed on Define External Quality Characteristics step), the evaluation team should define how this “new” quality characteristic will be evaluated through techniques available in literature or through one of the techniques represented in the Techniques Certification framework (presented on Chapter 4.3);

 Analysis of the Certification Techniques: The EMM level chooses on last step contains a set of evaluation techniques for each characteristic. The evaluation team, using their expertise, knowledge and know-how in those techniques must analysis such techniques and decide if those techniques are useful to evaluate the target software component or if it is necessary to define other techniques for executing / complement the evaluation. The idea is to define the best technique for each kind of evaluation. It is important to consider here that all of those techniques presented in EMM are recommended techniques to guide the evaluation team during the selection process, however, it is not avoided the

software component. Moreover, the evaluation team should justify the adoption of a new technique and if it is reasonable it can be incorporated into the EMM, looking for the incremental improvement of the model;

 Define Goal-Question-Metric (GQM): The evaluation team will define metrics to track the properties of the quality characteristics selected, the techniques adopted as well as the whole certification process. Through the Metrics Approach (described in session 4.4) the evaluation team will define:

o the metrics to evaluate those quality attributes selected from EQM or from external sources;

o the metrics necessary for each EMM technique used, which requires at least one metric for each evaluation technique that will be used; and

o the metrics to measure the efficiency of the component certification process;

The information collected using these metrics will support the quality assurance of the component evaluation and also provide insights for the next evaluations once they will be available on the Component Evaluation’s Repository;

 Establish Punctuation Level for Metrics: After defining all the metrics, the evaluation team should consider a punctuation level to facilitate its analysis. For example, if in a certain component evaluation if the metric defined to measure the Security quality attribute achieved less than 40%, it is not accepted; between 41% and 80% it could be considered reasonable; and higher than 81% it is considered acceptable and could receive the certification agreement. This punctuation level will be dependent on the importance of each quality characteristic (Define the Quality Characteristic step) and the evaluation level (Define EMM level step) defined during this activity. However, there is a kind of scale that should be

considered as basis in order to start the establishment of the punctuation level. The scale is defined on ISO/IEC 15504-2 (ISO/IEC 15504-2, 2000), as follows:

o N (Not Achieved): 0% to 15% - There is little or no evidence about the presence of the quality attribute on the component;

o P (Partially achieved): 16% to 50% - There is evidence about the presence of the quality attribute on the component. However, the quality attribute aspects have been partially achieved;

o L (Largely achieved): 51% to 85% - There is evidence about the presence of the quality attribute on the component. The component provides the quality attribute aspects implemented but it has not been completely achieved; and

o F (Fully achieved): 86% to 100% - The component provide all necessary fulfillments for the quality attribute in evaluation.

The evaluation team should analyze if the scale proposed on ISO 15504-2 makes sense to the component in evaluation. This is dependent on the reliability level expected by the component, i.e. the EMM level defined to evaluate the component. If necessary, the evaluation team must do some improvements in the scale proposed in order to become more accurate to the component in evaluation.

Moreover, the evaluation team should retrieve the previous evaluation in the Component Evaluation’s Repository in order to analyze the punctuation level for the metrics defined in previously evaluations;

 Develop the Specification of the Evaluation Document: The last step of this activity is elaborating a document with all theinformation collected during the Specify the Evaluation activity and generate the Specification of the

activity.