Software Measurement:

Software Measurement:

Software Measurement: Software Measurement:

Software Measurement:

Software Measurement:

Software Measurement: Software Measurement:

Software Measurement: The The The The The The The The Initial Climb

Initial Climb Initial Climb Initial Climb Initial Climb Initial Climb Initial Climb Initial Climb

Eduardo

Eduardo Santana de Almeida Santana de Almeida www.rise.com.br

www.rise.com.br

[email protected]

[email protected]

Outline Outline

Measurement

Some Definitions

The Measurement Process

Theory of Measurement

Software Measurement Programs

Empirical studies

Company reports

Current State

Out of Scope Out of Scope

Software Metrics Analysis

Software Metrics Tools

Software Measurement Standards

Software Measurement Processes

Measurement Measurement Measurement Measurement Measurement Measurement Measurement Measurement

is part of our is part of our is part of our is part of our is part of our is part of our is part of our is part of our

lives

lives

lives

lives

lives

lives

lives

lives

Medical Systems

Medical Systems

Atmospheric Systems

Atmospheric Systems

Avionic Systems

Avionic Systems

Map Systems Map Systems

Define Distance Choose a route

Measure

the speed

Predict

when arriving there

May 8, 2008 9/50

Measurement helps us Measurement helps us Measurement helps us Measurement helps us Measurement helps us Measurement helps us Measurement helps us Measurement helps us

to understand the to understand the to understand the to understand the to understand the to understand the to understand the to understand the

world, interact with it world, interact with it world, interact with it world, interact with it world, interact with it world, interact with it world, interact with it world, interact with it and improve our lives and improve our lives and improve our lives and improve our lives and improve our lives and improve our lives and improve our lives and improve our lives

[Fenton & Pfleeger, 1997]

[Fenton & Pfleeger, 1997]

Some Important Definitions Some Important Definitions

Measurement

Measurement is the process by which numbers or symbols are assigned to

attributes of entities in the real world in such a way to describe them according to clearly defined rules

Entity

Entity is an object (such as a person or a room) or an event (such as a journey or the testing phase of a software project) in the real world

Attribute

Attribute is a feature or property of an entity. Ex: area or color (room), cost (journey)

An Example An Example

“It is cold today” The air temperature is cold today

“He is taller than she” His height is greater that her height

[Fenton & Pfleeger, 1997]

[Fenton & Pfleeger, 1997]

* It is incorrect and unsuitable for scientific endeavors

** It is wrong to say that we measure things or that we measure attributes;

In fact, we measure attributes of things - “measure a room”

- length?

- area?

- temperature?

The Measurement Process The Measurement Process – –

Some Difficulties Some Difficulties

Person (what measure?) - Height

- Other attributes (intelligence) - IQ

Wine quality

Best soccer player

Accuracy (Measure) Instrument Measurement Definition

The Measurement Process The Measurement Process – –

Some Difficulties Some Difficulties

Is my scale acceptable?

- Height

- meters, inches, feet [person]

- miles, km [satellites]

[Fenton & Pfleeger, 1997]

[Fenton & Pfleeger, 1997]

Error Margins - Height

- Shoes - Posture

Which error margins are acceptable and which are not?

Measurement in Software Measurement in Software

Engineering Engineering

Donald Knuth - Structured Programming with go to Statements – ACM Computing Survey, Vol. 06, No. 04, 1974

The Reasons

- The “Big Picture” during software development - Is this product good| bad?

- Is my company developing good| better software?

- What kind of predictions can we do?

Neglect of Measurement in SE Neglect of Measurement in SE

- We fail to set measurable goals for software products - We fail to understand and estimate costs

- We cannot quantify or predict the quality of the products

- “I suppose that this agile process can be useful for our company”

- “Using that CASE tool our productivity can be improved by 70%”

Measurement in SE

Measurement in SE – – Cont. Cont.

-

Measures

are not defined in the context of well-defined measurement goal - The experimental hypothesis are often not made explicit

- Measurement definition do not always take into account the environment or

context

in which they will be applied

- A reasonable theoretical validation of the measure is often not possible because the attribute that a measure aims to quantify is often not well defined

- A large number of measure have never been subject to an empirical

validation

Measurement in SE

Measurement in SE – – Cont. Cont.

- Often

- Infrequently - Inconsistently - Incompletely - Frustration

- “70% sw costs involve maintenance. We can find 55 faults in 1000 LOC”

- How the results were obtained?

- What was measurable?

- What was the context?

Measurement in SE

Measurement in SE – – Cont. Cont.

- 2/3 metrics program fail in the first two years

- Fewer than 10% of the industry classified their attempt at implementing a metric program as positive

- Low incidence of successful metrics programs as being due to

organizational and managerial problems rather than short-

comings in the definitions of software metrics

Objectives for Software Objectives for Software

Measurement Measurement

- To assess the status of the project [product, process]

- To identify trends and corrective actions - To understand the project “Big Picture”

- To know how to “control” the project

- To understand what is happening during development - To control what is happening on our projects

- To improve our processes and products

It is about E N G I N E E R I N G

However, don

However, don ’ ’ t forget: t forget:

To manage the expectations before decisions

To show the margin of errors in

measurement

Software Measurement Software Measurement - -

Stakeholders Stakeholders

CEO

How satisfied are our clients?

How can reduce our costs?

Project Managers

What does each process cost?

How productive is my project team?

How much time can we fix defects?

Software Engineers

Was effective our code inspection?

Designers

Can my architecture be reusable?

Why Can be necessary a Theory?

Why Can be necessary a Theory?

- Hard questions for software engineering

- How much we know an attribute before it is reasonable to consider measuring it?

- complexity

- How do we know if we have really measured the attribute we wanted to measure?

- bugs -- quality

- What meaningful operations can we perform on measures?

We need:

- Rules to be followed

- To formalize the intuition about the way the world works

Real Life Real Life

- Iron Man is Taller than C-3PO - R2-D2 is not Tall

- Iron Man is Tall - C-3PO is Tall

Our approach for the real world is to do comparison

Real Life

Real Life – – cont. cont.

But, we can define relationships:

Given any two people, X and Y, we can observe that:

- X is Taller than Y, or - Y is Taller than X

“Taller than” is an empirical relationship for height

Theory of Measurement

Theory of Measurement – – cont. cont.

We can define another empirical relation on the same set:

- “Much Taller Than”

- Iron Man and C-3PO are much taller than R2-D2 - Empirical relations can be unary

- Iron Man is Tall

We can begin to understand the world by using relatively unsophisticated relationships

After initial understanding and some data, we can use more

sophisticated ways and tools

Theory of Measurement

Theory of Measurement – – cont. cont.

We can begin to understand the world by using relatively unsophisticated relationships After Initial understanding and some data, we can use more sophisticated ways and tools

Celsius scale 1742 AD

Fahrenheit scale 1720 AD

First thermometer measuring “hotter than”

1600 AD

Rankings, “hotter than”

2000 BC

Advances

Period

May 8, 2008 27/50

But, Why Do it?

- Age

- Eduardo’s age is 29

- Lee’s age is 39 // Lee is Chinese. Chinese include gestation time in their assessment of age

- Size (Lines of Code) - comments

- blank lines - libraries

More Definitions More Definitions

Measurement

Measure

A measure must specify the domain and range as well as the rules for performing the map

[Fenton & Pfleeger, 1997]

[Fenton & Pfleeger, 1997]

Key Stages of Formal Measurement Key Stages of Formal Measurement

- Identify attribute for some real world entities - Identify empirical relations for attributes

- Identify numerical relations corresponding to each empirical relation - Define mapping from real world entities to numbers

- Check that numerical relations preserve and are preserved by empirical relations

Real world Real world

C- C -3PO taller than R2 3PO taller than R2- -D2 D2 Number system Number system

M(C- M(C -3PO) > M (R2 3PO) > M (R2- -D2) D2) 28 28

10 10

May 8, 2008 29/50

Key Stages of Formal Measurement Key Stages of Formal Measurement

For the (binary) empirical relation “taller than”, we can have the numerical relation

x > y

Then, the representation condition requires that for any measure

M, A is taller than B if and only if M (A) > M(B)

For the (unary) empirical relation “is tall’, we might have the numerical relations

x > 27

The representation condition requires that for any measure M,

A is tall if and only if M (A) > 27

….

[Fenton & Pfleeger, 1997]

[Fenton & Pfleeger, 1997]

More Important Definitions More Important Definitions

Model

is an abstraction of reality, allowing us to strip away detail and view an entity or concept from a particular perspective

- Person (Height)

Predict system

consists of a mathematical model together with a set of

prediction procedures for determining unknown parameters and interpreting results

Scales

- Nominal - Ordinal - Interval - Ration - Absolute

Levels of Metrics Application Levels of Metrics Application

- Company or business unit level - Product group level

- Project level

- Component level

Software Measurement Programs Software Measurement Programs – – Why? Why?

• Process Improvement

• CMMI

• Decision Making

• Books (HP’s book, Fenton & Pfleeger)

• How to know how improvement efforts are actually affecting the

process/product

Software Measurement Programs Software Measurement Programs – – Data Data

• 10% industry classified a positive

• 2/3 did not last beyond second year

• Long and Complex

• Several Dilemmas

Software Measurement Software Measurement

Programs

Programs – – Ingredients Ingredients

• Empirical Models

• Technical issues

• Risks

• Culture

• Organizational issues

• Business issues

Software Measurement Frameworks Software Measurement Frameworks

• GQM

• M

³

P [Offen & Ross, 1997]

• Balanced ScoreCard [Becker & Bostelman, 1999]

• GQM|MEDEA – Metric Definition Approach [Briand et al., 2002]

• Nokiaway [Kilpi, 2001]

Software Measurement Software Measurement – –

Success Factors Success Factors

• Incremental Implementation

• Transparency

• Usefulness

• Develop Participation

• Metrics Integrity

• Feedback

[Hall & Fenton, 1997]

Software Measurement Nightmare Software Measurement Nightmare

• Most project engineers routinely collected some kind of tracking data, yet resisted the imposition of centralized data collection

• Project personnel expressed concern that the data collected would be used to assess individuals negatively

• Project leaders expressed concern about the time taken from sw development activities to collect metrics

• Project leaders expressed concern about how the collected data would

reflect poorly on their projects when compared to the data from other

projects

Measurement Program

Measurement Program – – How to How to access it

access it

• Metrics selection

• Collection effort

• Practitioner awareness

• Practitioner attitude

• Metrics integrity

Empirical Evidence Empirical Evidence

• Context

• 6 months

• 2 organizations

• Success Factors (organization’s willingness to)

• do background research on other metrics program and use advice given in the published experiential reports

• involve developers in metrics program design and inform them on the program development and progress

• use an incremental approach to implementation and run a pilot of the metrics program

Empirical Evidence

Empirical Evidence – – cont. cont.

•

Context

• 3 small companies

• Metrics program to evaluate how new practice and tools for configuration and change management were affecting the software process

• Part of a Process Improvement experiment by E.U’s European Systems and Software Initiative Program

• to motivate European sw companies to test and deploy best practices

• 18 months (commercial project)

• Companies

• 5 or fewer employees

[Kautz, 1999]

Empirical Evidence

Empirical Evidence – – cont. cont.

•

Important Factors

•

Management and commitment

• Project Planning and Organization

• Staff involvement and teamwork

• Education and Training

• Assessment, monitoring and evaluation of the new work practices and their technical support

• Adjust “rules of thumb” for the company environment/maturity

May 8, 2008 43/50

Empirical Evidence

Empirical Evidence – – cont. cont.

• Context

• to identify the various success factors on metrics programs success and study their direct and indirect effects on metrics program success

• Theoretical Model

• Technical factors

• metric collect, analysis

• metrics quality, tools

• dissemination

• incentives to make decision

• Organizational factors

• culture

• training

• feedback

• resource

• management support

• maturity level, institutional beliefs

[Gopal et al., 2002]

• Results

• To cultivate the culture

• First focus on technical factors and provide incentives for developers use metrics

Empirical Evidence

Empirical Evidence – – cont. cont.

• Empirical study

• Survey – December 1998 and February 1999 in conjunction with SEI

• 214 respondents - 126 companies

• Previous literature

• Longevity of the program

• Budget allocated to it

• This approach to measure “success”

• Use in decision making

• Impact on organizational performance

General Lessons Learned General Lessons Learned

• Based on controlled experiments

• 12 systems

• Lessons

• Many types of measurement goals (object| people)

• Models and Measures together

• Different types of measures

• Measurement-based analysis results are only good as the data they are based on

• Experimental approach

• Report specific measurement results in context

• Experiment replication

• Justify the measurement costs

• Tools

• Measurement can be used to predict maintenance

[Rombach, 1990]

General Lessons Learned

General Lessons Learned – – cont. cont.

• Experience based on consulting

• Lessons

• It is not about the metrics

• Success comes from channeling an organization’s pain into action

• Establishing a measurement program is easy; keeping it going is hard

• People skills matter more than quantitative skills

• Senior-level sponsorship and leadership are critical

• Measuring individuals can be okay (not consensus)

• Don’t go overboard trying to be perfect

Software Measurement Software Measurement Software Measurement Software Measurement Software Measurement Software Measurement Software Measurement Software Measurement –––– ––––

Practical Experiences

Practical Experiences Practical Experiences

Practical Experiences

Practical Experiences

Practical Experiences

Practical Experiences

Practical Experiences

Measurement Programs Measurement Programs - -

Practical Experiences

Practical Experiences – – cont. cont.

• U.S Army

• CECOM Program

• Goal

• To provide high-level management insight and awareness capabilities to address the proliferation of problems across the life-cycle of

mission-critical systems

• To investigate, develop, validate, and automate Army specific indicators and an application methodology and to transfer the technology for users

• Metrics to

• cost

• schedule

Measurement Programs Measurement Programs - -

Practical Experiences

Practical Experiences – – cont. cont.

•

U.S Army

•

Lessons Learned

• Improve with indicators in other projects

• Measurement was not an overhead (3% project budget)

• Difficult issue

• how to define lines of code

• how to measure manpower capabilities

[Fenick,1990]

Measurement Programs Measurement Programs - -

Practical Experiences Practical Experiences

• Motivation

• Publications in the literature

• Sw process improvement by engineers and managers

• Track productivity and quality

• Barriers

• “It is complex”

• Cultural issues

• Human issues

• Goals

• Discipline

May 8, 2008 51/50

Measurement Programs

Measurement Programs - - Practical Practical Experiences

Experiences – – cont. cont.

• Technical issues

• well-defined metrics

• guidelines for data collection and interpretation

• tools

• Metrics Working Group (MWG)

• 3 years to work – measurement program

• definition

• support

• deployment

• Lessons Learned

• Start with simple metrics

• Continuous improvement

• Sw and project management involvement

• Get involved software engineers and project managers to data analysis

• external consultant to begin it

• Culture

[Daskalantonakis, 1992]

Measurement Programs

Measurement Programs - - Practical Practical Experiences

Experiences – – cont. cont.

• Contel

•1988

• $3 billion Telecon company

• Contel Technology Center

• 13 members

• Metrics according to Project maturity

• Lessons Learned

• Starting improving the process

• Keep the metrics close to the developer

• Start with people who need help; then let them do your advertising for you

• Automate

• Keep things simple and easy to understand

Measurement Programs

Measurement Programs - - Practical Practical Experiences

Experiences – – cont. cont.

• Software Engineering Directorate (SED) of the Research, Development, and Engineering Center at the U.S. Army Missile Command

• Projects – 10.000 – 50.000 LOC

• Previous scenarios

• inaccurate and inconsistent data

• no validation

• no credibility

• Metrics Working Group (MWG)

• sw metric center, participation in standard groups

• define, collect, analyze, maintain the metric plan

• training

• Goal

• to evaluate productivity

• to calibrate the cost model

• Metrics

• effort, schedule, defects, size [Grable et al., 1999]

Measurement Programs Measurement Programs - -

Practical Experiences

Practical Experiences – – cont. cont.

•

Lessons Learned

• success – upper management support

• training

• experiment to validate and analyze the metrics

• periodic calibrate of the cost model

• initial-simple set of metrics

• feedback

• managers and engineers

• tools

May 8, 2008 55/50

Measurement Programs Measurement Programs - -

Practical Experiences

Practical Experiences – – cont. cont.

• Financial Software Solutions (FSS)

• Software Process Improvement Program

• Question

• How to implement a software metrics program to inform about the effect of SPI efforts within a specific company

[Iversen & Mathiassen, 2000]

• Lessons Learned

• Start simple

• Establish incentives

• Publish widely

• Feedback

• Use the data

Measurement Programs Measurement Programs - -

Practical Experiences

Practical Experiences – – cont. cont.

•

Nokiaway method

• DX200 Product Line

• 15 million of LOC

• + 2000 engineers

• Nokiaway Metric Program

• Metrics definition

• Data collection

• Metrics Analysis

• Metrics reporting

Software Measurement Software Measurement Software Measurement Software Measurement Software Measurement Software Measurement Software Measurement Software Measurement –––– ––––

Current Status

Current Status

Current Status

Current Status

Current Status

Current Status

Current Status

Current Status

Software Measurement

Software Measurement – – Current Current Status

Status

•

Status Report on Software Measurement

• [Pfleeger et al, 1997]

•

Software metrics: successes, failures and new directions

• [Fenton & Neil, 1999]

•

The State of Software Measurement Practice: Results of 2006 Survey

• [Kasunic, 2006]

Software Measurement

Software Measurement – – Current Current Status

Status

• Results

• State of the Gap

• researchers (much effort but it is being ignored by practitioners and customers) – practitioners – customers

• Gap between theory and practice

• Practitioners and customers know what they want, but the researchers have not yet been able to find measures that are practical, scientifically sound and cost effective to capture and analyze

• Researchers and Practitioners together

[Pfleeger et al, 1997]

Software Measurement

Software Measurement – – Current Current Status

Status

• Metrics are the same (1980)

• Based on LOC

• Assembly

• Java

• Defect counts

• Easy to collect| measure| understand

•

It can be considered success

• Books (+40)

• Journals

• Projects

• Conferences

May 8, 2008 61/50

Software Measurement

Software Measurement – – Current Current Status

Status

• It cannot be considered success

• There is no match in content between the increased level of metrics activity in academy and industry

• Much academic metrics research is inherently irrelevant to industrial needs

• scope

• content

• Much industrial metrics activity is poorly motivated

• Much industrial metrics activity is poorly executed

[Fenton & Neil, 1999]

Software Measurement

Software Measurement – – Current Current Status

Status

• Survey

• Software Engineering Measurement Analysis (SEMA) group at SEI

• 17 Questions

• Practitioners contacted SEI during 2004-2005

• More than 1000 answers

• 84 countries (53.1% U.S; 11.9% Europa; 8.4% India; 1.7% China)

• Goals

• the degree to which sw practitioners use measurement when conducting their work

• the perceived value of measurement

• approaches that are used to guide how measures are defined and used

• the most common types of measures used by software practitioners

“We believe that there is still much that needs to be done so that organizations use

Level of Involvement with Level of Involvement with

Measurement

Measurement

Are the Purposes for Are the Purposes for

Measurement Understood?

Measurement Understood?

Does Measurement Help Does Measurement Help

Performance?

Performance?

Is Measurement used to Is Measurement used to

Understand Quality?

Understand Quality?

Are documented Measurement Are documented Measurement

Processes used?

Processes used?

Are Measurement Definitions Are Measurement Definitions

Understood and Consistent?

Understood and Consistent?

Is Corrected Action Taken When Is Corrected Action Taken When

Thresholds are exceeded?

Thresholds are exceeded?

Measurement Guidanced Used

Measurement Guidanced Used

Measures That Are Reported

Measures That Are Reported

Concluding Remarks

Concluding Remarks Concluding Remarks

Concluding Remarks

Concluding Remarks

Concluding Remarks

Concluding Remarks

Concluding Remarks

Concluding Remarks Concluding Remarks

Measurement

Software Measurement Program

Empirical Results

Technical and Non-Technical issues

Long road for improvements

Similar issues with Software Reuse

Future Work Future Work

To investigate

Software Reuse Measurement Programs

Software Reuse Metrics

To define

Software Reuse Measurement Framework

To perform

Experimental Studies with the Framework

To define

Software Reuse Measurement Program

To perform

Experimental Studies with Software Reuse Measurement Program

References

References

References

References

References

References

References

References

References References

(Becker & Bostelman) S. A. Becker, M. L. Bostelman, Aligning Strategic and Project Measurement Systems, IEEE Software, Vol. 16, No. 03, May-June, 1999, pp. 46-51.

(Berry & Jeffery) M. Berry, R. Jeffery, An Instrument for Assessing Software Measurement Programs, Empirical Software Engineering Journal, Vol. 05, No. 03, November, 2000, pp. 183-200.

(Briand et al., 2002) L. C. Briand, S. Morasca, V. R. Basili, An Operational Process for Goal-Driven Definition of Measures, IEEE Transactions on Software Engineering, Vol. 28, No. 12, December, 2002, pp. 1106-1125.

(Clark, 2002) B. Clark, Eight Secrets of Software Measurement, IEEE Software, Vol. 19, No. 05, September-October, 2002, pp. 12-14.

(Darcy & Kemerer, 2005) D. P. Darcy, C. F. Kemerer, OO Metrics in Practice, IEEE Software, Vol. 22, No. 06, November-December, 2005, pp. 17-19.

(Daskalantonakis, 1992) M. K. Daskalantonakis, A Practical View of Software Measurement and Implementation

Experiences within Motorola, IEEE Transactions on Software Engineering, Vol. 18, No. 11, November, 1992, pp. 998-1010.

(Fenton & Neil, 1999) N. E. Fenton, M. Neil, Software metrics : success, failures and new directions, Journal of Systems and Software (JSS), Vol. 47, No. 2-3, July, 1999, pp. 149-157.

References References

(Fenick, 1990) S. Fenick, Implementing Management Metrics: An Army Program, IEEE Software, Vol. 07, No. 02, March, 1990, pp. 65-72.

(Gopal et al., 2002) A. Gopal, M. S. Krishnan, T. Mukhopadhyay, D. R. Goldenson, Measurement Programs in Software Development: Determinants of Success, IEEE Transactions on Software Engineering,

Vol. 28, No. 09, September, 2002, pp. 863-875.

(Gopal et al., 2005) A. Gopal, T. Mukhopadhyay, M. S. Krishnan, The Impact of Institutional Forces on Software Metrics Programs, IEEE Transactions on Software Engineering, Vol. 31, No. 08, August, 2005, pp. 679-694.

(Grable et al., 1999) R. Grable, J. Jernigan, C. Pogue, D. Divis, Metrics for Small Projects: Experiences at SED, IEEE Software, Vol. 16, No. 02, March-April, 1999, pp. 21-29.

(Hall & Fenton, 1997) T. Hall, N. Fenton, Implementing Effective Software Metrics Program, IEEE Software, Vol. 14, No. 02, March-April, 1997, pp. 55-65.

(Iversen & Mathiassen, 2000) J. Iversen, L. Mathiassen, Lessons from Implementing a Software Metrics Program, 33rd Hawaii International Conference on System Sciences (HICSS), Maui, Hawaii, U.S., January, 2000, pp. 7040-7041.

(Jeffery & Berry, 1993) R. Jeffery, M. Berry, A Framework for Evaluation and Prediction of Metrics Program Success, 1st International Software Metrics Symposium, IEEE Computer Society, Baltimore, Maryland, U.S., May, 1993, pp. 28-39.

(Kasunic, 2006) M. Kasunic, The State of Software Measurement Practice: Results of 2006 Survey, Technical Report, Software Engineering Institute (SEI), December, 2006, pp. 54.

References References

(Kautz, 1999) K. Kautz, Making Sense of Measurement for Small Organizations, IEEE Software, Vol. 16, No. 02, March-April, 1999, pp. 14-20.

(Kilpi, 2001) T. Kilpi, Implementing a Software Metrics Program at Nokia, IEEE Software, Vol. 18, No. 06, November-December, 2001, pp. 72-77.

(Li & Smidts, 2003) M. Li, C. S. Smidts, A Ranking of Software Engineering Measures Based on Expert Opinion, IEEE Transactions on Software Engineering, Vol. 29, No. 09, September, 2003, pp. 811-824.

(Maxwell & Forselius, 2000) K. D. Maxwell, P. Forselius, Benchmarking Software Development Productivity, IEEE Software, Vol. 17, No. 01, January-February, 2000, pp. 80-88.

(Maxwell, 2001) K. D. Maxwell, Collecting Data for Compatibility: Benchmarking Software Development Productivity, IEEE Software, Vol. 18, No. 05, September-October, 2001, pp. 22-25.

(Offen & Jeffery, 1997) R. J. Offen; R. Jeffery, Establishing Software Measurement Programs, IEEE Software, Vol. 14, No. 02, March-April, 1997, pp. 45-53.

(Pfleeger, 1993) S. L. Pfleeger, Lessons Learned in Building a Corporate Metrics Program, IEEE Software, Vol. 10, No. 03, May, 1993, pp. 67-74.

References References

(Rifkin, 2001) S. Rifkin, What Makes Measuring Software So Hard?, IEEE Software, Vol. 18, No. 03, May-June, 2001, pp. 41-45.

(Rombach, 1990) H. D. Rombach, Design Measurement: Some Lessons Learned, IEEE Software, Vol. 07, No. 02, March, 1990, pp. 17-25.