Copy of MIKKOshow - VTT project pages server

(1)

Slide 1 Copyright 2001, Fraunhofer IESE

Institut Experimentelles So f t w are Engineering Fraunhofer

IESE

Bernd Freimut

Fraunhofer Institute for Experimental Software Engineering Sauerwiesen 6

D-67661 Kaiserslautern Germany

Software Defect Measurement to Prevent Defects and Control Software Projects

Mikko

Mikko-Seminar, -Seminar, Oulu, Finland Oulu , Finland April 18, 2001 April 18, 2001

Fraunhofer IESE

• Software development as engineering discipline

• Establishment of „experimental“ learning cycles (observe – model – transfer – observe ...)

• Validation through measurement

• ca. 80 full-time employees

• Subsidiaries

– Kaiserslautern, Germany – FC-MD, Maryland, USA

Institute for Experimental Software Engineering

www.iese.fhg.de

(2)

IESE

IESE Core Competences

Core

competences reflect IESE corporate identity (active research areas)

• Experimentation

• Engineering-style Software Development

• Architecture-based Software Reuse

• Goal-oriented Software Modeling &

Quantification

• Experience Factory-based Learning

• Certifiable Education & Training in SE

Business Areas

• Software Development

(Quality SW Development, Rapid SW Development, Legacy System Reengineering, SD for Distributed Organizations, Component Ware, Product Line)

• Software Project Management

(Subcontractor Assessment & Mgt, Data driven Project/Quality/Risk Mgt, SW Procurement)

• Improvement Management

(Process Assessment & Improvement, Product Assessment & Improvement, IT Security)

• Software Competence Management

(Skill Profiling, SW Learning Organization, Job-oriented Education & Training, Education & Training on Demand)

• Software based Business Development (Innovation Management, ‘Expert reports’, SW Economics (e.g., ROI for LO), E-Business, Technology Consulting & Change Management)

Business areas reflect IESE focus on customer demands

(3)

IESE

Motivation for Software Defect Measurement

I can only think of one metric that is worth collecting now and forever: defect count. Any organization that fails to track and type defects is running at less than its optimal level Tom deMarco

Objective of this Presentation

• Which defect data can be collected and how can these data be analyzed in order to answer the following questions:

– How can we learn from our defects and prevent them in the future, resulting in lower development costs?

– How can we decide, whether inspections or tests have found an appropriate number and type of defects (assess and control V&V effectiveness)?

Defect Causal Analysis

Orthogonal Defect Classification

(4)

IESE

Summary

• This presentation showed you ...

– two practical techniques dealing with defects

§ Defect Causal Analysis

§ Orthogonal Defect Classification

– a process to improve your development process by including your qualitative reasoning about defects, their causes, and how to prevent them

– how to support your reasoning about defects with a sound, comprehensive defect classification scheme – how to analyze defect data to improve your

development process

– how to analyze defect data to control your projects ODC

DCA

In one hour ...

DCA

Defect Causal Analysis (DCA)

How can we learn from our defects and

prevent them in the future, resulting in

lower development costs?

(5)

IESE

Basic Idea of Defect Causal Analysis

• Prerequisites: defined basic development process, documented defects

• Analyze individual defects

• Trace back cause of defect

• Initiate and monitor actions to prevent defect occurrence in the future or to detect it earlier

Action Team Meeting Action

Team Meeting

Causal Analysis Meeting Causal Analysis Meeting

Defect Detection

Activity Defect Detection

Activity Software

Development Software Development

Defect Database

Defect Database Organization

Process Organization

Process

Software

sample of defects find defects fix defects

propose actions implement

actions base- line

DCA Activities

(6)

IESE

Causal Analysis Meeting

Causal Analysis

Meeting

Causal Analysis Meeting

• Developers analyze problems and recommend improvements at regular intervals

• Select sample of defects

§ Less than 20 representative defects

• Classify selected defects

§ When inserted, when detected, how fixed

• Identify systematic errors

§ An error that results in similar defects

• Determine root cause

§ Most important factor contributing to systematic error

• Develop action proposals

§ Prevent or detect earlier the systematic defect

• Document meeting results

*Causal Analysis Meeting (2)

• Agents: moderator, recorder, involved programmers, peer programmers

• Input: Defect database, classification scheme

• Output: Report

§ identification of meeting event

§ description of systematic errors

§ root cause for each systematic error

§ list of defects related to each systematic error

§ proposed actions (e.g., change coding standards, update inspection checklists,…)

(7)

IESE

Tool for Identifying Systematic Errors

Pareto Chart for defect type

Defect Type

Number of defects

0 1 2 3 4 5 6 7 8 9 10

Interface Data Logic Initialization Computation

Pareto Charts

Tool for Determining Root Cause

syst. error Methods

Tools

Input People

syst. error Methods

Tools

Input People

syst. error Methods

Tools

Input People

Cause-Effect Diagram

root cause

(8)

IESE

Action Team Meeting

• Software engineering process group with management support initiates actions

• Prioritize action proposals

§ Based on Pareto charts of causes, future development activities, relative ROI of actions

• Resolve conflicts and combine related proposals

§ Necessary for multiple causal analysis teams

• Establish implementation plan for high-priority items

• Motivate and justify changes

• Allocating resources and assigning responsibility for implementation plan

• Monitor progress of implementation and effectiveness of actions

• Ensure that success stories are recognized and individuals identified

*Action Team Meeting (2)

• Agents: software engineering process group with management buy-in

• Input: Reports from (multiple) causal analysis teams

• Output:

§ Implementation plan for actions

§ Progress report to management

ú may include statistics on defect distribution, action distribution by action category, cost of process

§ Progress report to causal analysis teams

ú developers should see that their suggestions are taken seriously

(9)

IESE

DCA Example

• Context:

– HP division developing systems software in C (SA/SD design, informal and sporadic code inspections, branch coverage testing, regression testing)

• Defects were collected during maintenance

• Defects were classified using a company-wide defect classification scheme

• Apply defect causal analysis as a regular practice – Use defect classification to identify problem areas – Check proposals with actual defects

DCA Example – Selecting Systematic Error

Logic Impl.

8%

Logic Descr.

5%

SW Interface 11%

Error Checking 16%

Documentation 12%

Specifications 19%

HW Interface

12% Module Design

17%

SW Interface 10%

Error Checking 15%

Module Design HW Interface 15%

11%

Specifications 39%

Logic Descr.

5%

Logic Impl.

Documentation3%

2%

Number of Defects Number of Defects

Normalized by Correction Effort

(10)

IESE

DCA Example – Brainstorming Root Causes

• Root Causes

– Unclear understanding of customer segments

– Lack of clearly assigned responsibilities – No document standards

– HW changes

• Improvement

– Marketing dep. sets up customer visits to learn about their needs

– Assign Config. Mgmnt. responsibility to one person (select tool for version control)

– Set up task force for new spec. format

*Evaluating DCA Process

Development stage

Number of defects per KLOC

CLD MLD Code UT/

FVT PVT/

SVT

AFTER BEFORE 4

8 12 16 20 24

(11)

IESE

DCA - Pro’s and Con’s

Pro’s

and

Con’s • Requires significant resources per defect (ca.

1.5% of project budget)

• Only a sample of all defects can be investigated

• Focusing on individual defects may lead to less attention to finding solutions addressing a larger scope of problem

• Helps improve both quality and productivity of organization

• Provides feedback at any stage of development process

• Helps show developers the value of conforming to process

Orthogonal Defect Classification (ODC)

How can we decide, whether inspections or tests have found an appropriate number and type of defects (assess and control V&V

effectiveness)?

&

How can software defect classification help us to learn from our defects?

ODC

(12)

IESE

Basic Idea of ODC

• Capture semantics of a defect by classification according to a few attributes (e.g., what had to be fixed, how was the defect detected, etc.)

• Analyze the actual distribution of attributes for a larger amount of defects

• Developers interpret deviations from expected distributions and

– Devise corrective actions in-process – Identify improvement actions

Defect ID

oAssignment xChecking oAlgorithm oFunction oInterface oTiming Component

Type Description

TriggerxBackward Compatibility oLateral Compatibility oDocument Consistency oOperational Semantics

ODC Activities

• Design of Classification Scheme – define attributes and their values – validate properties of scheme

• Data Collection and Classification

– collect and record defects found during inspections and testing

– assess attribute values for each defect (e.g., Defect Type, Trigger)

• Data Analysis and Interpretation

– evaluate whether inspection or test activities have achieved their purposes (monitor and control effectiveness)

– identify areas that may have an increased risk of field defects (where in the product, when are failures likely to occur)

– identify opportunities for process improvement (where should defects have been detected?)

(13)

IESE

Defect

Activity

When did you detect the defect?

Trigger

How did you detect the defect?

Impact What would have customer noticed if defect had escaped into the field?

Target

What high level entity was fixed?

Source

Who developed the target?

Age

What is the history of the target?

Defect Type What had to be fixed?

Defect Qualifier

Feedback to v&v process Feedback to product

Feedback to development process

[IBM]

Attributes of ODC Classification Scheme

Activity

Trigger

Impact

[IBM]

*Attribute Values for ODC Scheme (1)

• When did you detect the defect?

§ design inspection, code inspection, unit test, integration test, system test

• How did you detect the defect? What condition had to exist for the defect to surface? How to reproduce the defect?

– For each activity exists a different set of categories

§ Inspection Trigger: design conformance, logic/flow, lateral compatibility, backward compatibility, language dependency, concurrency, side effects, rare situation

• What would have customer noticed if defect had escaped into the field?

§ Capability, expected capability, installability, serviceability, standard, integrity/security, migration, reliability, performance, maintenance, usability, documentation

(14)

IESE

Target

Source

Age

Defect Type

Defect Qualifier

*Attribute Values for ODC Scheme (2)

• What high level entity was fixed?

§ requirements, design, code, build/package/merge, information, user-interface

• Who developed the target?

§ in-house, library, out-sourced, ported

• What is the history of the target?

§ base, new, rewritten, re-fixed

• What had to be fixed?

§ assignment, checking, algorithm, function, timing, interface, relationship

• (in combination with Defect Type)

§ missing, incorrect, extraneous

[IBM]

*Example Format of Defect Database

ID 1 2 3

open date 1.1.90 3.1.90 11.1.90 close date 2.1.90 10.1.90 13.1.90 activity inscpection unit test system test trigger conform. simple SW config impact capability usability reliability

target code code code

type assign checking function

qualifier misssing missing incorrect source in-house in-house outsourced

age new new refixed

(15)

IESE

Attribute Defect Type

Data Analysis Example:

How can we decide, whether inspections or tests have found an appropriate type of defects (assess V&V effectiveness)?

Defect

Activity

Trigger

Target

Source

Age

[IBM]

Attributes of ODC Classification Scheme

(16)

IESE

ODC-Attribute Defect Type

• Developer fixing the defect determines the defect type according to the nature of change

• Each defect type can be associated with a detection activity, in which it should be detected

– for each attribute value unique ‘signature’ (i.e., values change over time)

– for each detection activity unique distribution

Defect Type HLD LLD Code UT IT ST

Function: Requires Formal Design Change X

Assignment: Few LoC changed (e.g., initialization) X X Interface: Interaction with other components,modules X X Algorithm: Errors wrt. efficiency, correctness problems X X X Checking: No proper validation of data, loop conditions X X X

Timing/Serialization: problem with shared resources X X

Relationship: Associations among procedures, data structures X

Start Development

Start

Development ReleaseRelease

Development process

Defects

Design Inspections Unit Test Integration Test System Test

Function Assignment Interface Timing

Defect Types

%

10 20 30 40

Distribution tells us “where we are” in the process

Measuring Progress with Defect Type

Expected distributions over time

Actual distribution for one point in time

(17)

IESE

Percentage of defects found in Integration Test 4,35

39,13 10,67

17,78 1,19

Function Interface Checking Assignment Timing

0 5 10 15 20 25 30 35 40 45

Should have already been found

Should also detect these

Integration Test

Product entered integration test too early

Example in [Chaar, 1993]

Example: Analyze Defect Type

Attribute Trigger

Data Analysis Example:

How can we decide, whether inspections or tests have found an

appropriate type of defects (assess V&V effectiveness)?

(18)

IESE

Defect

Activity

Trigger

Target

Source

Age

[IBM]

Attributes of ODC Classification Scheme

Fault Failure

Trigger1

Trigger3 Trigger2

Defect life cycle

error->fault->activation->failure

ODC-Attribute Trigger

• Trigger activate and/or discover defects

• Inspector/Tester classifies according to condition that allows a defect to surface

§ What were you thinking about?

§ Why did you write the test case?

• Knowing the best triggers for specific defect types enables earlier detection

(19)

IESE

Inspection Trigger

Design Conformance (compare design with its spec) X X X X Understanding Details (of structure, operation) X X X Backward Compatibility (to previous version of product) X

Lateral Compatibility (to other systems/services) X

Concurrency (simultaneous use of resources) X

Logic Flow (operational semantics in question) X Side Effects (potential impact on another function/product) X X Documentation (code comments, user guides, manuals) X X X

Rare Situation (unusual sets of circumstances) X

Cross Products

Novice Within Product Expert

ODC-Attribute Trigger (for Inspections)

• Example: Inspection Trigger

– Each trigger value can be associated with a certain skill level of people

What did you think about?

First HLD-Inspection

Rare Situation Operational Semantic Backward Compatibili Document Consistency Design Conformance Lateral Compatibilit

0 10 20 30 40 50 60 70 80 90

expected:completeness and correctness addressed unexpected:too few interface and

lateral comp. for middleware

Algorithm Documentation Timing/Serialize Interface Function

High-Level Design Inspection

Evaluate Inspection Effectiveness (1)

(20)

IESE

Second HLD-Inspection with Experts

Rare Situation Operational Semantic Backward Compatibili Document Consistency Design Conformance Concurrency Lateral Compatibilit

0 5 10 15 20 25 30 35 40 45

Assignment Checking Build/Package Algorithm Documentation Timing/Serialize Interface Function

Many defects more

more defects more types

Second HLD-Inspection

Evaluate Inspection Effectiveness (2)

Attribute Trigger

Data Analysis Example:

How to analyze field defects to improve the development process?

(21)

IESE

Defect

Activity

Trigger

Target

Source

Age

[IBM]

Attributes of ODC Classification Scheme

Triggers for different activities

System Test

• Recovery

• Start / Restart

• Workload / Stress

• HW configuration

• SW configuration

• Normal Mode (block test)

Unit Test

• Coverage

• Sequencing

• Interaction

• Variation

• Simple Path

• Combination Path

Inspection

• Backward Compatability

• Lateral Compatability

• Design Conformance

• Logic Flow

• Side Effects

• Documentation

• Rare Situation

Field Defects

Normal Mode in field implies either an UT Trigger

an Insp. Trigger

(22)

IESE

0 20 40 60 80 100

Des.Insp Int.Test Syst.Test

Design Conformance Logic/Flow

Coverage Variation

Sequencing Interaction

Workload/ Stress Recovery/Exception HW Configuration SW Configuration

Improve Test Process through Trigger Analysis

• 120 field defects

• Questions from developers:

Where is improvement pontial?

– Int.-Test, Syst.-Test

• What can be improved?

– more Workload/Stress and SW Configuration test cases in system test

– improve test cases wrt.

coverage/variation in integration test

0 20 40 60 80

#Fehler pro Quartal

1Q 2Q 3Q 4Q 5Q 6Q 7Q 8Q

System Test Triggers

HW configuration Stress SW Configuration Recovery Startup

Understand Customer Usage through Triggers

• Different triggers have different usage profiles

• Similar profile in same product

• Validate system test against customer usage

§ Which trigger are detected?

§ Which are not detected?

• Customer-based service strategy

§ process improvements can be measured

§ insight in service and support demands

• Predictions of defect numbers possible

(23)

IESE

Defect

Activity

Trigger

Target

Source

Age

[IBM]

Attributes of ODC Classification Scheme

Improve Customer Satisfaction through Impact

• Analyse attribute Impact, to improve customer satisfaction

• Impact analysis shows major problem:

deficiencies with functionality

• Defect Type Analysis shows::

– >50% Missing Function hints to incomplete design or requirements – Algorithm defects hint to

poor Low-Level-Design

83%

1%

5%

3%0%1%0%4%3%

Capability Usability Performance

Reliability Installability Documentation Serviceability Security/Integrity Standards

0 50 100 150 200 250 300 350

Assignment Checking

Algorithm

/Method Function Interfa ce

Timing Relationship

Anzahl Fehler Incorrect

Missing

(24)

IESE

Pro’s and Con’s

Pro’s

and

Con’s • selection of attributes and values is empirical process

• defect measurement must cover most of the development process

• tool support for data collection is crucial for acceptance

• has not been tailored or used on early life cycle phases, e.g., requirements analysis

• provides in-process feedback to developers

• can also provide feedback on field defects

• managing defects at minimal cost (45sec- 2min/defect+Feedback Session)

• mathematically intuitive

Summary

• This presentation showed you ...

– two practical techniques dealing with defects

§ Defect Causal Analysis

§ Orthogonal Defect Classification

– a process to improve your development process by including your qualitative reasoning about defects, their causes, and how to prevent them

– how to support your reasoning about defects with a sound, comprehensive defect classification scheme – how to analyze defect data to improve your

development process

– how to analyze defect data to control your projects ODC

DCA

(25)

IESE

Questions?

Sure!

References

[1] David N. Card, Learning from our Mistakes with Defect Causal Analysis, IEEE Software, pp. 56--63, Jan. 1998.

[2] R.G. Mays, C.L. Jones, G.J. Holloway, and D.P. Studinski, Experiences with defect prevention, IBM Systems Journal, vol. 29, no. 1, pp. 4--32, 1990.

[3] James S. Collofello and Bakul P. Gosalia, Application of Causal Analysis to the Software Modification Process, Software -- Practice and Experience, vol. 23, pp. 1095-1105, Oct. 1993.

[4] O. Dangerfield, P. Ambardekar, P. Paluzzi, D. Card, and D. Giblin, Defect Causal Analysis: A report from the field, in Proceedings of the 2nd International Conference on Software Quality, ASQC, pp. 109--113, 1992.

[5] Ram Chillarege, Inderpal S. Bhandari, Jarir K. Chaar, Michael J. Halliday, Diane S. Moebus, Bonnie K. Ray, and Man-Yuen Wong, Orthogonal defect classification -- A concept for in-process measurements, IEEE

Transactions on Software Engineering, vol. 18, pp. 943--956, Nov. 1992.

[6] Kathryn A. Bassin, Theresa Kratschmer, and Padmanabhan Santhanam, Evaluating software development objectively, IEEE Software, vol. 15, pp.

66--74, Nov. 1998.

[7] Jarir K. Chaar, Michael J. Halliday, Inderpal S. Bhandari, and Ram Chillarege, In-Process Evaluation for Software Inspection and Test, IEEE Transactions on Software Engineering, vol. 19, pp.1055--1070, Nov. 1993.

Defect Causal Analysis

ODC

(26)

IESE

About ...

• This presentation is an excerpt from a larger tutorial given ...

– at several international conferences including ESCOM, Quality Week Europe, SPI

– for industrial customers of FhG IESE who are interested in software defect measurement

• Bernd Freimut is with the Fraunhofer Institute for Experimental Software Engineering. He is concerned with applied research and technology transfer in the areas of Software Measurement, Software Inspection Measurement, and Software Risk Management.

You can contact him under [email protected]

… the presentation

… the presenter

Backup-Slides

(27)

IESE

Trigger Distribution for Field Defects

0 20 40 60 80 100

Des.Insp Int.Test Syst.Test

Design Conformance Logic/ Flow

Coverage Variation

Sequencing Interaction

Workload/Stress Recovery/ Exception

HW Configuration SW Configuration

Trigger per quarter

0 20 40 60 80

#defects per quarter

1Q 2Q 3Q 4Q 5Q 6Q 7Q 8Q

System Test Triggers

HW configuration Stress SW Configuration Recovery Startup

(28)

IESE

Impact Distribution

83%

1%

5%

3%0%1%0%4% 3%

Capability Usability Performance

Reliability Installability Documentation Serviceability Security/Integrity Standards

Defect-Type x Defect Qualifier Distribution

0 50 100 150 200 250 300 350

Assignment Checking

Algorithm

/Method Function Interfa

ce Timing

Relati onship

#defects

Incorrect Missing