Value-based productivity measurement in software development projects

Texto

(1)Pós-Graduação em Ciência da Computação. “Value-Based Productivity Measurement in Software Development Projects” By. Gibeon Soares de Aquino Júnior Ph.D. Thesis in Computer Science. Universidade Federal de Pernambuco posgraduacao@cin.ufpe.br www.cin.ufpe.br/~posgraduacao. RECIFE, July 2010.

(2)

(3) UNIVERSIDADE FEDERAL DE PERNAMBUCO CENTRO DE INFORMÁTICA PÓS-GRADUAÇÃO EM CIÊNCIA DA COMPUTAÇÃO. GIBEON SOARES DE AQUINO JÚNIOR. “Value-Based Productivity Measurement in Software Development Projects". THESIS PRESENTED TO THE POST-GRADUATE PROGRAM IN COMPUTER SCIENCE OF THE UNIVERSIDADE FEDERAL DE PERNAMBUCO IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF PH.D. IN COMPUTER SCIENCE.. Adviser: Silvio Romero de Lemos Meira. RECIFE, July/2010.

(4) Aquino Júnior, Gibeon Soares de Value-based productivity measurement in software development projects / Gibeon Soares de Aquino Júnior. Recife: O Autor, 2010. xxv, 214 p. : il., fig., tab. Tese (doutorado) – Universidade Federal de Pernambuco. CIn. Ciência da Computação, 2010. Inclui bibliografia e apêndice. 1. Engenharia de software. 2. Medição. I. Título. 005.1. CDD (22. ed.). MEI2010 – 0105.

(5) This work is dedicated to my sons, João Victor and Luís Eduardo, and my wife, Silvia Helena..

(6)

(7) Acknowledgements • First of all, thanks to my family, for their comprehension and incentive in the most difficult moments of this journey. • I would like to thank Silvio Meira, my advisor, for his support and inspiration. • Thanks to the professors Jones Albuquerque, Paulo Borba and Renata Souza, for their support during this research. • Thanks to PROSE members, by the companionship during the studies and discussions about productivity. • Thanks to Sérgio Angelim, the C.E.S.A.R’s Operations Manager when my research started, by champion the C.E.S.A.R’s productivity improvament initiative and allow that i leaded the organizational efforts. • Thanks to Benedito Macêdo, the current C.E.S.A.R’s Operations Manager for allowing the use and publication of C.E.S.A.R’s project data. • Thanks to Izabella Lyra, my project manager during the crucial moments of this research, by the incentive and support, understanding my absense in important moments of the project where we worked. • Thanks to my C.E.S.A.R’s friends, by the feedbacks about the ideas and actions related to productivity measurement. • Thanks to C.E.S.A.R’s managers, by the participation on several key stages of this research.. vii.

(8)

(9) We don’t receive wisdom; we must discover it for ourselves after a journey that no one can take for us or spare us. —MARCEL PROUST (1871–1922).

(10)

(11) Resumo. A fim de melhorar a sua competitividade no mercado global, as organizações de software têm se preocupado cada vez mais com a questão de produtividade na execução de projetos. No entanto, para melhorar a produtividade, as organizações de software devem definir uma forma de medí-la. O problema é que a medição da produtividade apesar de parecer ser simples, sua aplicação concreta se mostra muito complexa. Muitos são os trabalhos de pesquisa sobre o tema, no entanto não há convergência sobre a métrica mais adequada de produtividade para as organizações de software. Baseado nos conceitos fundamentais relacionados à processos de produção, áreas de conhecimento social, evidências coletadas em organizações de software reais e análise do estado da arte em medição de produtividade em software, concluimos que a métrica mais adequada para medir a produtividade é específica para cada contexto organizacional, pois envolve estratégia, cultura organizacional, modus operandi, além de interesse e conhecimento daqueles diretamente envolvidos na medição e avaliação da produtividade. Isto explica porque não existe e nem há a possibilidade de existência de uma métrica de produtividade para projetos de software universalmente aceita. Baseado nestas descobertas, sugerimos a adoção de uma abordagem de medir produtividade baseada em valor. A hipótese central que orienta nossa trabalho de pesquisa é que uma abordagem baseada em valor pra medir a produtividade para medir a produtividade de projetos de software é mais adequada que as medições tradicionais. Uma das consequências da validade desta hipótese é que cada organização deve definir seu próprio modelo para a medição da produtividade. Com o objetivo de ajudar as organizações a definir e implementar um modelo próprio de medição de produtividade, um processo sistemático, com uma seqüência bem definida de etapas, entradas, saídas e diretrizes foi proposto. Ele envolve as atividades relacionadas com a definição, implementação e aperfeiçoamento do modelo de medição de produtividade. Além disso, foi baseado em uma extensa revisão dos principais desenvolvimentos relacionados com a medição da produtividade, além de ser influenciado por modelos de referência em engenharia de software, como IDEAL, CMMI, PSM e ISO/IEC 15939. O resultado da aplicação deste processo em uma organização de software produz um modelo de avaliação da produtividade, que considera a idéia de valor com base na visão dos principais stakeholders da organização. Finalmente, o conceito de medição de produtividade baseado em valor é adotado e avaliado em um estudo de caso, envolvendo em uma organização real de desenvolvimento de projetos de software. Em particular, o processo proposto para definição de modelos. xi.

(12) de medição de produtividade foi adotado e os benefícios, problemas e desafios foram avaliados com o objetivo de avaliar a eficácia do processo em atendar a o seu propósito. As análises do estudo de caso confirmaram que este tipo de abordagem foi de fato mais adequada para a organização estudada e que potencialmente pode ser aplicado a outras organizações de software. Palavras-chave: Medição de Produtividade, Medição baseada em valor, Melhoria de produtividade. xii.

(13) Abstract. With the aim of improving their competitiveness in nowadays global market, software organizations have been increasingly concerned with the issue of productivity in the execution of projects. However, to improve its productivity, software organizations must define a way to measure it. The problem is that productivity measurement is a great challenge, because while its concept is so simple, its concrete implementation is a very complex problem. Several are the research works about this theme, however there is no convergence about the most suitable measure of productivity for software organizations. Based on fundamental concepts of production processes, social knowledge fields, evidences collected in real software organizations and analysis of the state-of-art on productivity measurement, we concluded that productivity is a relative phenomenon, since different stakeholders perceive the result of the work differently and evaluate differently the outcomes of projects. For this reason, the most appropriate metric to measure the productivity is specific to each organizational context, because it involves organizational strategy and culture, modus operandi, beyond interest and knowledge of those directly involved in productivity measurement and evaluation. It explains why not exists and there is no possibility of existence of an universally accepted productivity metric in software projects. Based on these findings, we suggest adopting an approach to measure productivity based on value. The central hypothesis that guide our research is that a value-based approach to measure software projects productivity is more appropriate than traditional measurements. One consequence of the validity of this hypothesis is that each organization must define its own model for measuring productivity. In order to help organizations to define and implement their own productivity measurement model, a systematic process with a well-defined sequence of steps, inputs, outputs, and guidelines was proposed as part of this work. It involves activities related to the definition, deployment and improvement of the productivity measurement model. Furthermore, it was based on an extensive review of the key developments related to productivity measurement, besides being influenced by reference models in software engineering, as IDEAL, CMMI, PSM, ISO/IEC 15939. The result of applying this process in a software organization produces a model of productivity measurement that considers the most relevant outcomes, based on the vision of the main stakeholders in the organization. Finally, the concept of measuring productivity based on value was adopted and evaluated on a case study involving a real software organization. In particular, the proposed process to define productivity measurement models was used and the benefits, problems and challenges were assessed with the aim of evaluating the effectiveness of the. xiii.

(14) process to meet its purpose. The analysis of the case study confirmed that this approach was in fact more appropriate for the organization studied and that can potentially be applied to other software organizations. Keywords: Productivity Measurement, Value-based measurement, Productivity Improvement. xiv.

(15) Contents. 1. 2. 3. Introduction. 1. 1.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 2. 1.2. Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 5. 1.3. Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 6. 1.4. Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 8. 1.5. Statement of the Contributions . . . . . . . . . . . . . . . . . . . . . .. 9. 1.6. Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 11. Key Developments in the Field of Software Productivity Measurement. 13. 2.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 14. 2.2. The Genesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 15. 2.3. Productivity Measurement: A State Of The Art Review . . . . . . . . .. 16. 2.3.1. Physical Size Based Metrics . . . . . . . . . . . . . . . . . . .. 17. 2.3.2. Functional Size Based Metrics . . . . . . . . . . . . . . . . . .. 20. 2.3.3. Design Based Metrics . . . . . . . . . . . . . . . . . . . . . .. 22. 2.3.4. Multiple Size Based Metrics . . . . . . . . . . . . . . . . . . .. 23. 2.3.5. Summary of Productivity Measurements . . . . . . . . . . . . .. 25. 2.4. Related State Of The Art Reviews . . . . . . . . . . . . . . . . . . . .. 30. 2.5. Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 31. 2.6. Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . .. 32. Software Productivity Measurement Foundations. 35. 3.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 36. 3.2. Productivity Measurement in Software Projects . . . . . . . . . . . . .. 36. 3.2.1. Software Production Process . . . . . . . . . . . . . . . . . . .. 37. 3.2.2. Common Mistake about software productivity measurement . .. 39. 3.2.3. Measuring Software Productivity . . . . . . . . . . . . . . . .. 41. 3.2.4. Practical Considerations . . . . . . . . . . . . . . . . . . . . .. 44. Software Productivity Relativity . . . . . . . . . . . . . . . . . . . . .. 46. 3.3.1. A Discussion About Software Productivity Relativity . . . . . .. 47. 3.3.2. Practical Evidences of Software Productivity Relativity . . . . .. 49. 3.4. Productivity Dysfunction in Software Projects . . . . . . . . . . . . . .. 50. 3.5. Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . .. 52. 3.3. xv.

(16) 4. Foudations of a Process to Measure Productivity in Software. 55. 4.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 56. 4.2. Process Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 56. 4.3. Process Main Elements . . . . . . . . . . . . . . . . . . . . . . . . . .. 57. 4.3.1. Phases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 58. 4.3.2. Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 58. 4.3.3. Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 60. 4.3.4. Work Products . . . . . . . . . . . . . . . . . . . . . . . . . .. 61. Reference models relationships . . . . . . . . . . . . . . . . . . . . . .. 62. 4.4.1. IDEAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 62. 4.4.2. Practical Software Measurement . . . . . . . . . . . . . . . . .. 66. 4.4.3. ISO/IEC 15939 Standard . . . . . . . . . . . . . . . . . . . . .. 68. 4.4.4. CMMi Measurement and Analysis . . . . . . . . . . . . . . . .. 69. 4.4.5. Other Models . . . . . . . . . . . . . . . . . . . . . . . . . . .. 71. Process Adoption Considerations . . . . . . . . . . . . . . . . . . . . .. 74. 4.4. 4.5 5. 3ProMiSE’s Specification. 77. 5.1. Startup phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 78. 5.1.1. Establish the context . . . . . . . . . . . . . . . . . . . . . . .. 78. 5.1.1.1. Define the goal . . . . . . . . . . . . . . . . . . . . .. 79. 5.1.1.2. Define the scope . . . . . . . . . . . . . . . . . . . .. 80. 5.1.1.3. Identify the stakeholders . . . . . . . . . . . . . . . .. 81. Plan the definition . . . . . . . . . . . . . . . . . . . . . . . .. 81. 5.1.2.1. Develop a high-level plan . . . . . . . . . . . . . . .. 82. 5.1.2.2. Establish commitments . . . . . . . . . . . . . . . .. 82. Definition phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 83. 5.2.1. Define perspectives . . . . . . . . . . . . . . . . . . . . . . . .. 83. 5.2.1.1. Identify the perspectives . . . . . . . . . . . . . . . .. 84. 5.2.1.2. Prioritize and merge the perspectives . . . . . . . . .. 86. 5.2.1.3. Specify the perspectives . . . . . . . . . . . . . . . .. 87. Identify dimensions . . . . . . . . . . . . . . . . . . . . . . . .. 88. 5.2.2.1. Identify the relevant outputs . . . . . . . . . . . . . .. 89. 5.2.2.2. Identify the relevant inputs . . . . . . . . . . . . . .. 90. 5.2.2.3. Assess the relevance of the inputs and outputs . . . .. 91. 5.2.2.4. Define Metrics . . . . . . . . . . . . . . . . . . . . .. 91. 5.2.2.5. Evaluate the measurement costs . . . . . . . . . . . .. 93. 5.1.2. 5.2. 5.2.2. xvi.

(17) 5.2.2.6. Prioritize metrics . . . . . . . . . . . . . . . . . . .. 93. Define the model . . . . . . . . . . . . . . . . . . . . . . . . .. 94. 5.2.3.1. Define the measurement model . . . . . . . . . . . .. 95. 5.2.3.2. Evaluate and Adjust the model . . . . . . . . . . . .. 96. Deployment phase . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 96. 5.3.1. Plan the adoption . . . . . . . . . . . . . . . . . . . . . . . . .. 96. 5.3.1.1. Define the Plan . . . . . . . . . . . . . . . . . . . .. 97. 5.3.1.2. Approve the Plan . . . . . . . . . . . . . . . . . . .. 98. 5.3.2. Deploy the model . . . . . . . . . . . . . . . . . . . . . . . . .. 98. 5.3.3. Evaluate the deployment . . . . . . . . . . . . . . . . . . . . .. 99. 5.2.3. 5.3. 6. Integrating 3ProMiSE with multicriteria approaches 6.1. Multicriteria Problem in 3ProMiSE . . . . . . . . . . . . . . . . . . . . 102. 6.2. Solving 3ProMiSE’s problem with Data Envelopment Analysis . . . . . 103 6.2.1. 7. 101. DEA Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 103 6.2.1.1. Background . . . . . . . . . . . . . . . . . . . . . . 103. 6.2.1.2. DEA’s Basic Model . . . . . . . . . . . . . . . . . . 104. 6.2.1.3. Numeric Example . . . . . . . . . . . . . . . . . . . 106. 6.2.1.4. DEA advanced use . . . . . . . . . . . . . . . . . . . 108. 6.2.2. DEA Justification . . . . . . . . . . . . . . . . . . . . . . . . . 110. 6.2.3. DEA-3ProMiSE Integration . . . . . . . . . . . . . . . . . . . 111. 6.2.4. DEA-3ProMiSE Integration Limits . . . . . . . . . . . . . . . 112. The 3ProMiSE adoption in C.E.S.A.R: A Case Study 7.1. 115. Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 7.1.1. Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116. 7.1.2. Justification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117. 7.1.3. Our Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 7.1.3.1. Determine and define the research questions . . . . . 118. 7.1.3.2. Design the case study . . . . . . . . . . . . . . . . . 118. 7.1.3.3. Prepare to collect the data . . . . . . . . . . . . . . . 119. 7.1.3.4. Collect data in the field . . . . . . . . . . . . . . . . 119. 7.1.3.5. Evaluate and analyze the data . . . . . . . . . . . . . 120. 7.1.3.6. Prepare the report . . . . . . . . . . . . . . . . . . . 120. 7.2. Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120. 7.3. Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121. xvii.

(18) 7.4. 3ProMiSE Use Description . . . . . . . . . . . . . . . . . . . . . . . . 122 7.4.1. 7.4.2. 7.5. 7.4.1.1. Establish the context . . . . . . . . . . . . . . . . . . 123. 7.4.1.2. Plan the definition . . . . . . . . . . . . . . . . . . . 123. 7.4.1.3. Define perspective(s) . . . . . . . . . . . . . . . . . 124. 7.4.1.4. Identify dimension(s) . . . . . . . . . . . . . . . . . 125. 7.4.1.5. Define the model . . . . . . . . . . . . . . . . . . . . 134. 7.4.1.6. Plan the Adoption . . . . . . . . . . . . . . . . . . . 135. 7.4.1.7. Deploy the model . . . . . . . . . . . . . . . . . . . 135. 7.4.1.8. Evaluate the deployment . . . . . . . . . . . . . . . 136. Second Experience . . . . . . . . . . . . . . . . . . . . . . . . 137 7.4.2.1. Establish the context . . . . . . . . . . . . . . . . . . 137. 7.4.2.2. Plan the definition . . . . . . . . . . . . . . . . . . . 137. 7.4.2.3. Define perspectives and dimensions . . . . . . . . . . 138. 7.4.2.4. Define the model . . . . . . . . . . . . . . . . . . . . 139. 7.4.2.5. Plan the adoption . . . . . . . . . . . . . . . . . . . 140. 7.4.2.6. Deploy the model . . . . . . . . . . . . . . . . . . . 140. 7.4.2.7. Evaluate the deployment . . . . . . . . . . . . . . . 140. Case Study Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 7.5.1. 7.5.2. 8. First Experience . . . . . . . . . . . . . . . . . . . . . . . . . 123. Conclusions. Research Questions Analysis . . . . . . . . . . . . . . . . . . . 142 7.5.1.1. Answers for RQ1 . . . . . . . . . . . . . . . . . . . 143. 7.5.1.2. Answers for RQ2 . . . . . . . . . . . . . . . . . . . 145. 7.5.1.3. Answers for RQ3 . . . . . . . . . . . . . . . . . . . 148. General Critical Analysis . . . . . . . . . . . . . . . . . . . . . 150 7.5.2.1. Construct validity . . . . . . . . . . . . . . . . . . . 150. 7.5.2.2. Internal Validity . . . . . . . . . . . . . . . . . . . . 151. 7.5.2.3. External Validity . . . . . . . . . . . . . . . . . . . . 151. 7.5.2.4. Reliability . . . . . . . . . . . . . . . . . . . . . . . 151 153. 8.1. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154. 8.2. Research Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 156. 8.3. Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157. 8.4. Future Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158. 8.5. Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . 159. xviii.

(19) Appendices. 163. A 3ProMiSE’s Templates 163 A.1 Perspectives Specification . . . . . . . . . . . . . . . . . . . . . . . . . 164 A.2 Dimensions Specification . . . . . . . . . . . . . . . . . . . . . . . . . 165 A.3 Metric Specification (Measurement Plan) . . . . . . . . . . . . . . . . 166 B C.E.S.A.R’s Dimensions Specification. 167. C C.E.S.A.R’s Metrics Specification 173 C.1 Productivity Measurement Plan - First Experience . . . . . . . . . . . . 174 C.2 Productivity Measurement Plan - Second Experience . . . . . . . . . . 177 D C.E.S.A.R’s Projects Data 181 D.1 Project Data from First Experience . . . . . . . . . . . . . . . . . . . . 182 D.2 Project Data from Second Experience . . . . . . . . . . . . . . . . . . 182 E Case Study Questionnaire 183 E.1 Questionnaire for evaluation of outputs and inputs relevance for C.E.S.A.R.184 E.2 Answers from the questionnaire after be applied in C.E.S.A.R. . . . . . 191 F Universities Data From Frontier Analyst Tool. 193. xix.

(20)

(21) List of Figures. 1.1. Software Costs Trends (Boehm, 1987). . . . . . . . . . . . . . . . . . .. 2. 1.2. Examples of software demand growth. . . . . . . . . . . . . . . . . . .. 3. 1.3. Parts of Software Productivity Problem. . . . . . . . . . . . . . . . . .. 4. 1.4. Our Ph.D. research steps. . . . . . . . . . . . . . . . . . . . . . . . . .. 7. 1.5. ProSE Solutions Framework. . . . . . . . . . . . . . . . . . . . . . . .. 10. 2.1. Software productivity metrics studies distribution over decades. . . . . .. 29. 2.2. Acumulated distribution of software productivity metrics studies. . . . .. 30. 3.1. Simple Production Process. . . . . . . . . . . . . . . . . . . . . . . . .. 37. 3.2. Simplified Production Process. . . . . . . . . . . . . . . . . . . . . . .. 39. 3.3. Composite Production Process. . . . . . . . . . . . . . . . . . . . . . .. 39. 3.4. Production process with multiples perspectives. . . . . . . . . . . . . .. 47. 3.5. The dysfunction phenomenon (Adapted from (Austin, 1996)). . . . . .. 51. 4.1. 3ProMiSE Basic Structure. . . . . . . . . . . . . . . . . . . . . . . . .. 57. 4.2. 3ProMiSE Phases . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 58. 4.3. 3ProMiSE Activities. . . . . . . . . . . . . . . . . . . . . . . . . . . .. 60. 4.4. IDEAL Model (Adapted from (McFeeley, 1996)). . . . . . . . . . . . .. 63. 4.5. 3ProMiSE to IDEAL Mapping. . . . . . . . . . . . . . . . . . . . . . .. 65. 4.6. PSM Process Model (Adapted from (Florac et al., 1997)). . . . . . . . .. 67. 4.7. 3ProMiSE to PSM Mapping. . . . . . . . . . . . . . . . . . . . . . . .. 68. 4.8. 3ProMiSE to ISO/IEC 15939 Mapping. . . . . . . . . . . . . . . . . .. 69. 4.9. Measurement & Analysis Process Area of CMMi (Adapted from (Goldenson et al., 2003)). . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 70. 4.10 3ProMiSE to MA-CMMi Mapping. . . . . . . . . . . . . . . . . . . .. 72. 4.11 PDCA Process Model . . . . . . . . . . . . . . . . . . . . . . . . . . .. 73. 4.12 DMAIC Process Model. . . . . . . . . . . . . . . . . . . . . . . . . .. 73. 4.13 BSC Perspectives Process Model. . . . . . . . . . . . . . . . . . . . .. 74. 5.1. Startup Phase’s Workflow. . . . . . . . . . . . . . . . . . . . . . . . .. 78. 5.2. Definition Phase’s Workflow. . . . . . . . . . . . . . . . . . . . . . . .. 83. 5.3. An example of software project perspectives. . . . . . . . . . . . . . .. 85. 5.4. Deployment Phase’s Workflow. . . . . . . . . . . . . . . . . . . . . . .. 96. xxi.

(22) 6.1 6.2. Regression Line vs Frontier Line. Adapted from (Cooper et al., 2000). . 105 DEA integration into 3ProMiSE. . . . . . . . . . . . . . . . . . . . . . 111. 7.1 7.2 7.3 7.4 7.5 7.6 7.7 7.8. C.E.S.A.R’s Organizational Structure. . . . . 3ProMiSE’s First Cycle Experience. . . . . . Inputs and Outputs identified by perspective. . C.E.S.A.R’s outputs measures. . . . . . . . . C.E.S.A.R’s inputs measures. . . . . . . . . . 3ProMiSE’s First Adoption Experience. . . . 3ProMiSE’s Second Cycle Experience. . . . . 3ProMiSE’s Second Adoption Experience. . .. . . . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . .. 121 124 129 131 133 135 138 140. A.1 Indicator Specification in a Measurement Plan. . . . . . . . . . . . . . 166 A.2 Metric Specification in a Measurement Plan. . . . . . . . . . . . . . . . 166 C.1 C.2 C.3 C.4 C.5 C.6 C.7 C.8 C.9 C.10 C.11 C.12. xxii. Productivity Metric Specification. . . . Functional Ratio Metric Specification. . Profit Ratio Metric Specification. . . . . Client Satisfaction Metric Specification. Team Satisfaction Metric Specification. Cost Ratio Metric Specification. . . . . Productivity Metric Specification. . . . Functional Size Metric Specification. . . Client Satisfaction Metric Specification. Team Satisfaction Metric Specification. Total Cost Metric Specification. . . . . HR Capacity Metric Specification. . . .. . . . . . . . . . . . .. . . . . . . . . . . . .. . . . . . . . . . . . .. . . . . . . . . . . . .. . . . . . . . . . . . .. . . . . . . . . . . . .. . . . . . . . . . . . .. . . . . . . . . . . . .. . . . . . . . . . . . .. . . . . . . . . . . . .. . . . . . . . . . . . .. . . . . . . . . . . . .. . . . . . . . . . . . .. . . . . . . . . . . . .. . . . . . . . . . . . .. . . . . . . . . . . . .. . . . . . . . . . . . .. 174 174 175 175 176 176 177 178 178 179 179 180.

(23) List of Tables. 2.1. Putnam and Myers Skill Factors Table. . . . . . . . . . . . . . . . . . .. 19. 2.2. Software productivity metrics studies . . . . . . . . . . . . . . . . . . .. 25. 5.1. Specification of Establish Context activity. . . . . . . . . . . . . . . . .. 79. 5.2. Specification of Plan Definition activity. . . . . . . . . . . . . . . . . .. 81. 5.3. Specification of Define Perspectives activity. . . . . . . . . . . . . . . .. 84. 5.4. Perspectives Priority Scale. . . . . . . . . . . . . . . . . . . . . . . . .. 87. 5.5. Team’s Perspective Specification. . . . . . . . . . . . . . . . . . . . . .. 88. 5.6. Specification of Identify Dimensions activity. . . . . . . . . . . . . . .. 89. 5.7. Outputs and inputs relevance scale. . . . . . . . . . . . . . . . . . . . .. 92. 5.8. Team’s Dimensions Specification example. . . . . . . . . . . . . . . .. 92. 5.9. Measurement Adoption Facility. . . . . . . . . . . . . . . . . . . . . .. 94. 5.10 Specification of Define the Model activity. . . . . . . . . . . . . . . . .. 95. 5.11 Specification of Plan the Adoption activity. . . . . . . . . . . . . . . .. 97. 5.12 Specification of Deploy the Model activity. . . . . . . . . . . . . . . .. 98. 5.13 Specification of Evaluate the Model activity. . . . . . . . . . . . . . . .. 99. 6.1. Single input and single output Cooper’s case. . . . . . . . . . . . . . . 104. 6.2. Inputs and Outputs from Universities Example. . . . . . . . . . . . . . 107. 6.3. Efficiency score of Universities. . . . . . . . . . . . . . . . . . . . . . 108. 7.1. Perspectives Specification. . . . . . . . . . . . . . . . . . . . . . . . . 126. 7.2. Metrics Adoption Facility. . . . . . . . . . . . . . . . . . . . . . . . . 132. 7.3. Metrics Typical Values. . . . . . . . . . . . . . . . . . . . . . . . . . . 134. 7.4. Weights used to productivity calculus. . . . . . . . . . . . . . . . . . . 134. 7.5. Productivity Calculus of C.E.S.A.R’s projects. . . . . . . . . . . . . . . 136. 7.6. New Productivity Index of C.E.S.A.R’s projects. . . . . . . . . . . . . . 141. 7.7. Effort spent by C.E.S.A.R in 3ProMiSE adoption. . . . . . . . . . . . . 146. 7.8. Results from questionnaires. . . . . . . . . . . . . . . . . . . . . . . . 147. 7.9. Productivity Data Comparison. . . . . . . . . . . . . . . . . . . . . . . 147. 7.10 Spearman’s Rank Correlation Coefficient . . . . . . . . . . . . . . . . 148 7.11 Statistical Significance of Correlation Results . . . . . . . . . . . . . . 148 A.1 Template for Perspectives Specification. . . . . . . . . . . . . . . . . . 164 A.2 Dimensions Specification Template. . . . . . . . . . . . . . . . . . . . 165. xxiii.

(24) B.1 B.2 B.3 B.4. Organizational Dimensions Specification. Client Dimensions Specification. . . . . . Product Dimensions Specification. . . . . Team Dimensions Specification. . . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. 168 169 170 171. D.1 Gross collected data by 2008’s Experience. . . . . . . . . . . . . . . . 182 D.2 Gross collected data by 2009’s Experience. . . . . . . . . . . . . . . . 182 E.1 Compilation of questionnaire answers. . . . . . . . . . . . . . . . . . . 191 F.1. xxiv. Universities Example Data . . . . . . . . . . . . . . . . . . . . . . . . 194.

(25) List of Abbreviations. C.E.S.A.R. Centro de Estudos e Sistemas Avançados do Recife (Recife Center for Advanced Studies and Systems). BSC. Balanced Scorecard. CEO. Chief Executive Officer. DSI. Delivery Source Instruction. ELOC. Executable Lines of Code. ERP. Enterprise Resource Planning. ESLOC. Effective Source Lines of Code. IFPUG. International Function Point Users Group. NCSS. Non Commenting Source Statements. NESMA. Netherlands Function Point Users Group. PMO. Project Management Office. SLOC. Source Lines of Code. UCP. Use-case Point. xxv.

(26)

(27) 1 Introduction “One step forward and you are no longer in the same place.” —CHICO SCIENCE (1966–1997). This chapter presents the general problem of productivity for software development. In particular, it explains the motivation for this work and details the problem of measuring productivity in software projects. Furthermore, it details the objectives of this work and describes the research methodology adopted. Finally, the organization of this document is presented.. 1.

(28) CHAPTER 1. INTRODUCTION. 1.1. Motivation. Productivity is considered one of the most important instruments to improve the competitiveness and profitability of an industry (Porter, 1998). The global market requires that companies increasingly improve their quality levels while reduce their production costs. The main way to reduce production costs, specifically in software development, is through productivity improvement, i.e., developing more with less cost or effort. For this reason, productivity improvement has been a constant concern for software organizations. Due to the demand growth for software products, the necessity and importance of improving productivity in the software development is increasing over years. Gibbs (1994) reports that the size of these software systems is doubling every 5 to 10 years. According to Boehm (1987) the worldwide costs with software is increasing continuously at a rate of 12% globally, as presented in Figure 1.1.. Figure 1.1 Software Costs Trends (Boehm, 1987).. According to Merlyn et al. (1994), software importance is also rapidly increasing in mission critical projects. Two examples in Figure 1.2 show the software demand growth in complex fields, such as military aircraft and space flight (Humphrey, 2001; Boehm, 1987). Figure 1.2(a) shows the software demand growth in military aircraft between 1960 e 2000. The F-4, in 1960, had 8% of its functions controlled by software. With the F-16, in 1982, this number reached 45%. In 2000, on the F-22, 80% of everything that the pilot does is controlled by software. Figure 1.2(b) shows the software demand growth at NASA programs, since Mercury in 1962 until the Space Station in the early 1990’s. The Mercury Project had about 1,500,000 object instructions to support it, while Space 2.

(29) 1.1. MOTIVATION. Station Project had approximately 80,000,000 object instructions.. (a) Software growth (Humphrey, 2001).. in. military. aircraft (b) Growth in software demand for U.S space flight (Boehm, 1987).. Figure 1.2 Examples of software demand growth.. Many advances have been done to improve productivity in software development with the research, development, standardization and dissemination of good practices for that purpose (Pietrasanta, 1968; Aron, 1969, 1976; Boehm, 1981; Boehm et al., 1982; Vosburgh et al., 1984; Jones, 1986; DeMarco and Lister, 1987; Maxwell et al., 1996; Boehm et al., 2000; Jones, 2000). However, the truth is that there is no definitive solution that can be applied to solve all the problems related to software development (Brooks, 1978) and software productivity improvement has no difference. In software context there is not a systematic view of what can be done to improve productivity and how the investments can be prioritized or organized in order to incorporate productivity throughout the software development cycle, in a secure, planned and gradual manner (Boehm, 1987). Furthermore, Groth (2004) claims that one of the factors responsible for this problem is the lack of an industry wide standard definition for software productivity. Different studies interpret the productivity concept in particular way and for this reason propose distinct solutions, and in some cases contradictories. The wide literature about this theme indicates that there was a great evolution related to what we know about software productivity, however it also presents several challenges in this field (Scacchi, 1989; Maxwell et al., 1996). Several questions still reside and there are many myths about the software productivity. In particular, about what is its meaning, how to measure it, what affects it and how to improve it. Figure 1.3 shows these specific problems and specify how they are structured in terms of dependency. In the figure, any problem situated below serves as foundation for the problem above. For this reason, the fundamental problem is related to the meaning of productivity in software development. The next problem, which can be addressed when the previous has been treated, is the. 3.

(30) CHAPTER 1. INTRODUCTION. productivity measurement and so on.. Figure 1.3 Parts of Software Productivity Problem.. The productivity problem segmentation, presented by the Figure 1.3 was performed as part of this research, as will be explained in Section 1.3. However, our reseach work will address only the productivity measurement problem and, consequently, the meaning of productivity problem. This couple of problems represent a concise and relevant challenge because it is the foundation for any practical research in software productivity. Dealing only with the meaning of productivity problem represent exclusively a theoretical scope. In another way, approaching the productivity measurement problem is both a practical and a conceptual challenge, but it also requires to address the meaning of productivity problem since the latter is a requirement for the former. About the productivity measurement theme, we detected that there is no convergence about the most suitable measure of productivity for software organizations, as will be viewed in Section 2.3. The simplest and most commonly used metric is the SLOC/ManMonth and some of its variations, but they are so problematic for this purpose because its use is sensible to a lot of problems related to language syntax (Jones, 1978) and program formatting (Boehm, 1987). Moreover, Multiple Size metrics, which involves assessment of productivity in many dimensions, show a good alignment with the idea of producing value and are recently gaining popularity, but they are more complex, less common and are still maturing. According to Boehm (2003); Boehm and Huang (2003) quantifying the outcomes of what is produced in a software project should not be reduced to the requirements, code quantities and complexity, but the product value to the stakeholders. An ideal metric of productivity should be able to measure how effectively and efficiently ideas are turned into software products (Duncan, 1988). For this reason, the productivity metrics based 4.

(31) 1.2. PROBLEM STATEMENT. on value, which consider multiple dimensions and correspond to what the stakeholders consider as the value being produced, is the most complete and accurate for modern organizations.. 1.2. Problem Statement. Much progress has been made in the field of software productivity measurement (Scacchi, 1989; Maxwell et al., 1996; Aquino and Meira, 2009b,c), however several issues already exists in this field. There are many divergences about productivity measurement in software projects. Nowadays, there is no direct nor safe answer for the problem about the ideal productivity metric. On the other hand, there is a trend toward the use of metrics that tries to incorporate the produced-value idea, as will be discussed in detail in Chapter 2. Boehm and Huang (2003) argues that the traditional used metrics represents an over-simplification of the real problem, they do not measure the produced-value to the stakeholders and they are not nearby of the business organization perspective. The typical approach of productivity measurement fails to measure the productivity in producedvalue viewpoint and create a measurement model that does not correctly address what is established by the strategic objectives of the organization, as shown in Section 3.2. Moreover, this approach may cause a problem known as Dysfunction (Austin, 1996), which can be explained as the occurrence of a phenomenon where the way how the team works to achieve a target defined by the organization causes a fall in actual performance, and this occurrence is not reflected in the measurements used (This concept will be expatiated in Section 3.4). An explanation for this problem is that may not be possible to derive an universal model for measuring productivity for any type of software organization and projects. Each organization must define its own model for productivity measurement, which can even vary depending on the types of projects that are executed (Chapter 3 discusses it in details). Based on this idea, we propose that productivity is a relavive concept. This phenomenon was called of Law of Software Productivity Relativity, specified by Law 3.1, and its veracity will be discussed in details in Section 3.3. This research work takes this law as assumption and based on it produces a chain of contributions in order to prove the hypothesis that a value-based approach to measure software projects productivity is more appropriate than traditional measurements. For this reason we can declare our main research question as:. 5.

(32) CHAPTER 1. INTRODUCTION. Law 1.1. Is the value-based productivity measurement more appropriate than traditional measurements for software projects? In order to answer this question, more effort needs to be made toward the definition, systematization and validation of new ways of measuring the productivity, which considers the value-based idea for a specific organization. Thus, the main goal of this work, what justifies its originality, can be stated as: This works has as aim at defining and validating a systematic way to perform the definition and deployment of productivity measurement model based on the producedvalue in software organizations, according to the stakeholders viewpoint and the strategic perspective.. 1.3. Methodology. This work took place in an environment where theory and practice coexists in a synergic relationship. The general problem emerged from an industrial context, while the theory proposed was applied in practice and the results were used to improve the theory. This kind of research method is best known as Action Research (Brydon-Miller et al., 2003), which according to Baburoglu and Ravn (1992) has the aim to solve current practical problems while expanding scientific knowledge. Moreover, it is oriented to action, where the researcher is concerned with the creation of organizational change and, simultaneously, studying the process. Vezzosi (2006) states that Action Research method is not a linear one, but rather a interplay between practice, reflection and learning, whose main feature is the cyclical, recursive nature. This research work started simultaneously with an organizational initiative whose aim was increase the productivity of running software projects. This organization is a medium scale Brazilian software development company, called C.E.S.A.R1 . The steps of our research methodology can be viewed in Figure 1.4. Initially, the macro-scope of the research includes the aim of defining solutions to improve the company’s productivity. For this reason, the first step was to investigate state-of-art in the software productivity matter. After this review we segmented the problem in four main subjects, as represented by the Figure 1.3, and decided that the first two subjects already represent great challenge to be investigated as the focus of our Ph.D. research, as explained in Section 1.1. 1 http://www.cesar.org.br. 6.

(33) 1.3. METHODOLOGY. Figure 1.4 Our Ph.D. research steps.. With the scope of the research defined, we review the existing productivity metrics and measurement studies published in literature (in Chapter 2), with the aim of understanding the current progress in this theme and identifying open issues. Some findings were identified, particularly that there is no convergence about the most suitable measure of productivity for software organization, and that a special category of productivity measurement (Multiple Size) is a more suitable for this purpose, besides its is becoming popular over decades. In order to explain the absence of a widely accepted measure of productivity for software projects, we investigate the conceptual foundations of this problem and concluded that productivity is a relativity phenomenon and for this reason it needs to be defined and analyzed only in a particular context. Hence an adaptive measurement category, as in the Multiple Size, is more useful in software development context. Finally, based on the fact that each organization must defined its own productivity measurement model, we defined a process with the aim of supporting the definition and the deployment of productivity measurement model in software projects (as described in Chapter 4 and Chapter 5). The first version of the process was defined and its applicability was tested in C.E.S.A.R environment. Several needs for improvement were identified and a new version of the process was specified and applied again there. The details and results of this experience were reported in Chapter 7. During this case study, the process and the measurement model resultant from its execution was validated in terms of practicality, validity, and benefits for the accuracy of productivity measurement.. 7.

(34) CHAPTER 1. INTRODUCTION. 1.4. Context. This work is part of a framework of solutions related to software productivity improvement which is the scope of work of a research group called ProSE (Productivity in Software Engineering)2 . Its main aim is investigating and developing the state-of-the-art related to productivity in software engineering. The problem that concerned this framework is how to treat productivity improvement as something which may be planned, controlled and executed, independent from common software process improvement models, as CMMi (Chrissis et al., 2003). Its goal is to define guidelines that focus on productivity improvement and enable a software organization to be more competitive, implementing only the best practices essential to enhance its production capacity. Historically, productivity improvement was treated as consequence of quality improvement programs based on maturity models, statistical control and others. Instead, we are proposing a new approach in which productivity is the central focus and software process improvement is only one of the methods to achieve it. Figure 1.5 shows an overview of how the productivity problem was structured in order to allow the creation of specific solutions in the productivity field. The productivity problem is segmented in the following areas: • Productivity Improvement Strategies – The purpose of this research area is to propose ways of improving the productivity of software organizations through definitions of strategies, which compile the good and bad practices already documented about the theme. It is also concerned with the investigation, discovery and documentation of new ways of improving productivity in software organizations. Moreover, its aim is giving a systemic view about the correlation and trade-offs related to these strategies; • Reward systems – This area, which is subpart of the Productivity Improvement Strategies, addresses the specific problem of defining and implementing incentive systems in software development organizations with the goal of improving the productivity of teams on projects. In particular, this theme aims to define strategies through the creation of solutions to guide managers in the implementation of incentive systems, also known as reward systems. The proposed solutions will focus on supporting initiatives to improve productivity, establishing ways for the organization to measure some aspects related to team productivity, and so rewarding 2 http://prose.cesar.org.br.. 8.

(35) 1.5. STATEMENT OF THE CONTRIBUTIONS. them in some way that are suitable for the specific organizational context, for example, through financial recognition, promotions, awards and benefits (Furtado et al., 2009a,b); • Productivity factors – The general purpose of this area is investigating and documenting factors, which have impact on software development productivity, besides defining the influence level of each one of the specific types of software organizations, such as software factories, software-houses, IT department inside non-software organizations, etc. Thus, a knowledgeable body of factors influencing the productivity, which will help organizations to decide where to invest their resources in order to achieve its specifics goals, may be constructed; • Productivity Metrics (scope of this work) – This area aims to define the most suitable productivity metrics for each organizational context. In particular, its purpose is to help software organizations to define its productivity measurement model in a way that makes it near to the organizational strategic level. Moreover, it considers the idea of produced-value in the productivity metrics (This idea will be best explained in Chapter 3); • Productivity Measurement (scope of this work) – The aim of this area is to define systematic ways of establishing and sustaining productivity measurement initiatives in software organizations. It compiles the best practices already documented about software measurement in general and adapt them to the specific requirements of productivity measurement in organizations, besides investigating and experimenting new practices in this scope; • Infrastructure and tools for Productivity Measurement – This area has an aim of defining and constructing infrastructure and tools to help the definition, deployment and maintenance of initiatives of productivity measurement in software organizations.. 1.5. Statement of the Contributions. As result of the work presented in this thesis, the following contributions can be enumerated:. 9.

(36) CHAPTER 1. INTRODUCTION. Figure 1.5 ProSE Solutions Framework.. 1. An extensive study of the key developments in the field of software productivity measurement, in an attempt to analyze this research area and, mainly, identify trends to follow (Aquino and Meira, 2009b); 2. The definition of a taxonomy of productivity metrics, with the aim of group those with similar characteristics and organize its diversity in a coherent manner; 3. The formalization of the productivity measurement problem (called Law of Productivity Relativity) and its evaluation based on theoretical argumentation and analysis of practical evidences collected in a real software organization (it will be viewed in Chapter 3); 4. The definition and formalization of a process for productivity measurement, using the approach of produced-value, in software projects (Aquino and Meira, 2009a). The proposed process defines and systematizes the steps of the productivity measurement life cycle, involving the start-up of the initiative, definition of a productivity measurement model and its deployment in organization. It will be viewed in details in Chapter 4 and Chapter 5; 5. The planning and operation of an initiative of adopting the proposed process in a 10.

(37) 1.6. OUTLINE. real software organization. Moreover, the analysis and interpretation of its results and, consequently, evaluation of the viability of the proposed process for software productivity measurement.. 1.6. Outline. The remaining of this document is organized as follows: • Chapter 2 conducts an investigation about the productivity metrics and measurement studies published in literature, categorize the productivity metrics identified according to the type of measure used and identify the main lessons learned about this matter. Based on literature review and on analyses of proposed metrics found, we propose some directions in order to evolve the state of the art on software productivity measurement and, consequently, on the software productivity theme; • Chapter 3 brings a discussion about the possibility of the existence of a universally accepted productivity metric in software projects. Moreover, based on concepts of production process, social knowledge fields and evidences collected in real software organizations it proposes and argues about the Law of Software Productivity Relativity; • Chapter 4 presents an overview of a process to support the definition of productivity measurement models in software organizations which considers the challenges of value-based productivity measurement. Particularly, the process definition was divided in two chapters, the current chapter has the aim of introducting the foundations of the process, besides mapping its activities to elements of well-known models in software measurement; • Chapter 5 represents the second part of the process, where the activities of the process to support the definition of productivity measurement models in software organizations are detailed. Moreover, the process is specified in a formal manner as a set of activities, organized in phases, that produces outputs (work products) and are performed by specific roles; • Chapter 6 describes how multi-criteria approaches may be used to complement the 3ProMiSE. In particular, it explains and justify the use of Data Envelopment Analysis technique inside the 3ProMiSE process.. 11.

(38) CHAPTER 1. INTRODUCTION. • Chapter 7 describes a case study of the use of the proposed process in a Brazilian software company and evaluate its results; • Chapter 8 summarizes the contributions of this work, presents the final conclusions and discuss opportunities for future works;. 12.

(39) 2 Key Developments in the Field of Software Productivity Measurement. “Not to know what has been transacted in former times is to be always a child. If no use is made of the labors of past ages, the world must remain always in the infancy of knowledge.” —CICERO (106 BC–43 BC). This chapter conducts an investigation about the productivity metrics and measurement studies published in literature, categorizes the productivity metrics identified according to the type of measure used and identifies the main lessons learned about this matter. Based on the literature review and on analysis of metrics found, we propose some directions in order to evolve the state of the art on software productivity measurement.. 13.

(40) CHAPTER 2. KEY DEVELOPMENTS IN THE FIELD OF SOFTWARE PRODUCTIVITY MEASUREMENT. 2.1. Introduction. In order to improve productivity it is necessary, first, to understand how to measure it. DeMarco (1982) emphasizes that the first step toward improvement and control is to measure, as his statement says: “You can’t control what you can’t measure”. Therefore, this chapter in particular, will explore the already proposed methods to measure productivity reported in literature. Productivity is conceptually a simple measure, and it is defined as the ratio between what is produced and what is consumed: productivity = out put/input. The concept of productivity is widely used in many areas of expertise, from agriculture to economics. Despite the idea of productivity being very simple and popular in other areas, specifically in software development, it is a historically complex problem. According to Jones (1978) the lack of a precise and non-ambiguous definition of productivity metric has been a source of problems in software projects. This difficulty in measuring the productivity in software development is not caused by the productivity itself, but by the intrinsic characteristic that is the meaning of out put in software projects context. Quantifying the produced results in a software project in terms of size, complexity or value to the customer is a very complex problem (Boehm, 1987). Moreover, depending on how and which indicators are measured the conclusions about the productivity can be completely different (Scacchi, 1989). The extensive literature on productivity and productivity measurement confirms the scientific interest on this subject. Many issues are still without answers and there are still many myths about the problem of productivity. This chapter investigates several studies published in literature on metrics and measurement of productivity in software development projects in order to raise the main lessons learned on this matter. In particular, the Section 2.2 describes the first discussions and studies published in literature about the productivity improvement and measurement. The Section 2.3 describes the studies about productivity measurement (a literature review) and categorizes them according to the type of measurement used (Sections 2.3.1 to 2.3.4). Moreover, we identified some trends in the literature review (Section 2.6), analyzed some related works (Section 2.4) and proposed some future directions in software productivity measurement research to improve the state of the art in this subject (Section Section 2.5). 14.

(41) 2.2. THE GENESIS. 2.2. The Genesis. Productivity has been one of the most discussed issues in Software Engineering, both in academic and industrial fields. A great variety of studies addressing this issue and much progress in the state of the art has already been made. But today, after several studies published on the matter, there is a lot of problems that remain open, and the industry and academia are still looking for new ways of achieving productivity gains in software development. Despite the emphasis on productivity nowadays, this problem is not new. At the beginning of Software Engineering in 1968, NATO Conference, where about 50 experts in computing, from 11 countries gathered in Garmisch, Germany, to discuss the problems of software development at the time (Naur and Randell, 1969), the concern with productivity was already mentioned. Even during the discussions were cited several factors affecting productivity, such as programming language, programmer experience, culture and motivation, which are still well supported by recent studies in the area. Additionally, previous studies have confirmed the importance of the problem. The first empirical studies, based on statistical data analysis from the U.S. Army software development projects, appeared in the mid-60 in the SDC (System Development Corporation) (Farr and Zagorski, 1964; Farr and Nanus, 1964). They had the goal of identifying factors affecting the cost of software projects and, as consequence, improving the process of estimation. Although these works did not have the direct purpose of productivity analysis, they can be considered part of this scope since they measured the projects productivity, besides identified several factors affecting productivity. Later, several studies (Farr et al., 1965; Farr and Zagorski, 1965; Weinwurm and Zagorski, 1965; Nelson, 1966; Fleishman, 1966; La Bolle, 1966), in the same area at the System Development Corporation, have extended the previous work exploring the problem with data from new projects and based on the results of the practical application of previous results. Furthermore, LaBolle (1966) published a cost estimation model, as result of an extensive study done by the investigation of 169 finalized projects. This study analyzed various factors that had influence on the final development effort, or more specifically, on the productivity of software projects. Other more specific studies about this problem have also emerged over time from the IBM’s Federal Systems Division (FSD) (Pietrasanta, 1968; Aron, 1969, 1976). Most of them were part of a great search for better estimation methods, which allowed a better reliability in predicting the costs of software projects and thus avoiding the typical. 15.

(42) CHAPTER 2. KEY DEVELOPMENTS IN THE FIELD OF SOFTWARE PRODUCTIVITY MEASUREMENT. problem of this area, known as the Software Crisis (Naur and Randell, 1969). Throughout the Software Engineering history, many studies have been conducted related to the problem of productivity and as could be seen, this matter has already been discussed since 60’s. Although there are several aspects of the problem that are still open, there are, on the other hand, a large range of relevant results which indicate the possibilities of exploration of this issue. Therefore, this area of research is very promising and relevant from the scientific viewpoint.. 2.3. Productivity Measurement: A State Of The Art Review. Measuring productivity of projects has been a major challenge for practitioners and researchers in software development. Despite the difficulty of effectively measuring the productivity of software projects, much effort has been made in an attempt to obtain ways to assess productivity, following the philosophy of Gilbs’s law (Gilb, 1977) which state that “Anything you need to quantify can be measured in some way that is superior to not measuring it at all”. As already seen in Section 2.2, the main proposals of productivity metrics emerged with the first studies on software productivity. These studies used metrics based on SLOC, as a measure of out put. Over the time, other proposed measures were defined and used as an alternative, but nowadays there is no single productivity measure that is well accepted by scholars and practitioners of the matter. Due to the large number of studies, using different productivity metrics, it is essential to group them according to some criteria in order to investigate them effectively. As part of this state of the art review, a taxonomy of productivity metrics was created. This taxonomy defines groups, according to the approach for measuring the out put. For this reason, the studies reviewed in this section shall be organized according to this taxonomy, as shown below: • Physical Size – It considers the studies using SLOC, variations or derivations. Examples of SLOC variations are Delivery Source Instructions, Executable Lines Of Code and Developed Statements. The SLOC derivations are calculated from the source code, as such Non-Commented Source Code and Effective Source Lines of Code. These studies will be presented in Section 2.3.1; • Functional Size – It considers the studies using output measures based on number 16.

(43) 2.3. PRODUCTIVITY MEASUREMENT: A STATE OF THE ART REVIEW. of features perceived by the user. The most common examples of these types of metrics are Function Points and Use-case Points. Such studies will be presented in Section 2.3.2; • Design Size – In this group, which will be presented in Section 2.3.3, are the studies that used metrics related to the design of the project, counting the software size according to the structure of the code. The most common examples of metrics of this type are number of modules, packages or classes; • Multiple Size – The studies of this group, described in Section 2.3.4 use a more accurate viewpoint to assess the productivity, using multidimensional models which assess different aspects of what is produced in a software project.. 2.3.1. Physical Size Based Metrics. Although SLOC is something quite basic, it is widely used in productivity studies as the out put measure in the productivity calculus (productivity = out put/input). In studies that followed, we found that different metrics are used for output, but all based on the physical lines counting. Different names are used to represent the same concept, such as Delivery Source Lines, Machine Instructions, Delivery Source Instruction and Executable Lines Of Code. Another difficulty encountered when compiling these studies was that in most of them there is no precise specification of what is considered in the counting of these metrics (e.g., comments, blank lines, statements, etc.). For this reason, further analysis and comparison between them becomes more difficult. Initial studies on this matter (Farr et al., 1965; Nelson, 1966) widely used a metric similar to SLOC, predecessor of SLOC per man-month, but with the same principle, called Man-Months per Machine Instructions, or more specifically Man-Months per 10.000 Machine Instructions. Aron (1969, 1976), also used metric of SLOC in studies on projects at IBM. Wolverton (1974) conducted an empirical research on TRW’s projects and used Object Instructions per Man-Month to measure productivity. Subsequently, Walston and Felix (1977) conducted a study in the broader IBM Federal Systems Division, under the data of 60 projects, and used DSL/MM as a metric of productivity, where DSL (Delivery Source Lines) means only those lines of code delivered as part of the product and MM the total effort in man-months (Man-Month). At the end of the ’70s, Jones (1978) reported that productivity and quality metrics, in terms of lines of code are paradoxical, since SLOC per unit of effort tends to emphasize more the size and format of the programs than their efficiency and quality of these.. 17.

(44) CHAPTER 2. KEY DEVELOPMENTS IN THE FIELD OF SOFTWARE PRODUCTIVITY MEASUREMENT. According to him, this metric tends to penalize more high-level languages than languages such as Assembly. For this reason the SLOC is not a good indicator of economic productivity, despite being frequently used. Despite this criticism, in the ’80s, Lawrence (1981), with the aim of evaluating the productivity of developers, used the metric of SLOC/Hours in 278 projects developed in Australia. In the same year, Bailey and Basili (1981) in their proposal of a software estimation meta-model used a metric SLOC/Man-Month to measure productivity in projects in the SEL (Software Engineering Laboratory). Still in the same decade, Boehm (1981); Boehm et al. (1982) had done an extensive use of metric DSI/MM in the productivity assessment and software costs estimation models. Vosburgh et al. (1984) studied the productivity in large-scale projects and used the metric of developed statement/personyear. Grady and Caswell (1987a) used the metric of NCSS3 as out put and Person-Month as input to evaluate the productivity of projects in HP. Similarly, Card et al. (1987) used the productivity as being Developed Non-Comment SLOC/Programmer-Hours (where the metric Developed Non-Comment SLOC was similar to NCSS) to analyze the effects of certain practices on productivity of 22 scientific projects in the SEL. Althought, Boehm had widely used this type of measurement, he agreed with Jones (1978) that this metric may be paradoxical and showed that a well-formatted program could have three times its original size (Boehm, 1987). Focusing on the problem of the input metric, Arthur (1985), on the other hand, proposes the use of a productivity metric which better reflects the prospect of financial input. In this work he proposes the use of the metric ELOC/$ as a measure of productivity, where the ELOC means Executable Lines of Code. In the ’90s, a larger study about software projects productivity from U.S. and Japan was performed (Cusumano and Kemerer, 1990) using the metric of non-comment SLOC as an out put metric and (work-years) as the input metric. Later, Banker et al. (1991) performed a study in 65 maintenance projects of a large banking systems in the U.S. using the metric of SLOC as a measure of size. Blackburn et al. (1996) also use SLOC as out put measure and considers only the coding effort to measure the input. Briand et al. (1998) used the metric NCSS/person month, excluding the code produced by code generators to evaluate the productivity of projects in a study that had as objective the identification of correlation between the overhead and productivity. Chidamber et al. (1998) used the metric of SLOC and hours as measures of out put and input, respectively, 3 It is a metric well known and well defined,. of comments and white spaces.. 18. which is based on SLOC and calculated through the removal.

(45) 2.3. PRODUCTIVITY MEASUREMENT: A STATE OF THE ART REVIEW. in a study on the use of the object oriented metrics for management. In the same period, in an initiative which took place at AT&T Bell Laboratories with the aim of establishing and improving the measurement of software productivity, Yu et al. (1991) used a metric called Net developed code. This metric include the new, modified and bug lines of code, as out put measure, while the input was measured as person-year (considering only technical people effort). Putnam and Myers (1991) also proposed, a new metric called Process Productivity (Equation 2.1). According to them is superior to the standard productivity metric based on SLOC, despite being based on it. This productivity measure was based on years of experience in project management of R&D in the U.S. armed forces, and later in General Electric Co. and involved projects developed in the United States, Britain, Australia and Japan. Process Productivity =. Size 1. 4. ( E f βf ort ) 3 × Time 3. (2.1). where, • Size - Is the product size in ESLOC (Effective Source Lines of Code); • β - Is a factor called Skill Factor, which is the function of project size, according to Table 2.1; • E f f ort - Is the total project effort in person-years; • Time - Is the schedule of project in years.. Size (SLOC) 5-15K 20K 30K 40K 50K > 70K. β 0.16 0.18 0.28 0.34 0.37 0.39. Table 2.1 Putnam and Myers Skill Factors Table.. In despite of this, Maxwell et al. (1996) confirmed that the standard metric SLOC is better than the Process Productivity in a study of European Space Agency projects database, with 99 projects in 8 European countries.. 19.

(46) CHAPTER 2. KEY DEVELOPMENTS IN THE FIELD OF SOFTWARE PRODUCTIVITY MEASUREMENT. Nowadays, studies on productivity also make use of the SLOC metric, such as MacCormack et al. (2003) that widely used SLOC by person-day on 29 software projects to evaluate the trade-offs of process improvement on the quality and productivity. Reifer (2002) also used the metric SLOC/Sta f f -Month to evaluate the productivity in a database with data from 500 projects from 38 organizations in 12 different fields of applications. Moses et al. (2006) used two variations of the productivity metric, they were Hours/SLOC and SLOC/Sta f f -Month, in a case study comparing productivity of projects in an organization and in the study of Reifer (2002) projects. But according to them, the metric of Hours/SLOC is more reliable for comparative purposes, since the concept of Sta f f Month is not standardized among different studies and benchmarks. Dilorenzo et al. (2008) also used SLOC measures for productivity measurement and effort estimation in C.E.S.A.R. As could be viewed, the metrics based on SLOC are still commonly used to evaluate the productivity of software projects, despite the problems indicated by Jones (1978, 1986) and others (Albrecht, 1979; Boehm, 1987; Duncan, 1988; Rosenberg, 1997; Faulk et al., 2004). One of the explanations for this phenomenon, besides the SLOC simplicity, is that the most popular effort estimation models in software engineering (Boehm, 1981; Panlilio-Yap, 1992; Boehm et al., 2000) also use it as input variable.. 2.3.2. Functional Size Based Metrics. This section considers the studies which use out put measures based on number of features perceived by the user. The most common examples of these metrics type are FP (Function Points) (Albrecht, 1979) and UCP (Use-case Points) (Karner, 1993). In an attempt to define a metric which takes the functional size of the system into consideration, and is not influenced by aspects such as type of the programming language adopted, the code structure and other aspects which were not related to the user’s viewpoint, Albrecht (1979) developed a technique to quantify the size of the software whose unit of measure was the Function Point. The main objective of this measure was to count the size and complexity of the system according to the viewpoint of the user. One of the characteristics of this counting technique was that the size of the system being developed was independent of technology, programming language used or even technical decisions such as choice of algorithms or data structures. This measure was initially used to assess productivity in 24 different projects at IBM, ranging from 3,000 to 318,000 SLOC developed between 1974 and 1978. On the one hand, Behrens (1983) used metrics based on function point to measure 20.

(47) 2.3. PRODUCTIVITY MEASUREMENT: A STATE OF THE ART REVIEW. the productivity in 25 different projects developed between 1970 and 1980, with the size ranging between 27 and 599 function points and the effort between 600 and 28,700 hours. More specifically he used the metric of productivity as hours/FP, where FP is the number of function points. Banker et al. (1991) conducted studies in 65 large banking systems maintenance projects in the U.S. using the SLOC metric and the function point. In the same business, Parkan et al. (1997) also used metrics based on function point to evaluate the performance of development teams in a large bank in Hong Kong, as part of a reward system. Jones (1991) and Moses et al. (2006) also used this metric as out put measure and man-month as input. However, in both cases the function point was calculated from the SLOC, using conversion factors by programming language. Symons (1988, 1991) criticizes the use of the original function points and suggests an alternative to this metric. The metric and the proposed method of counting are intended to address the shortcomings of the original method and consequently its measure. The new metric was called “Mark II” Function Point, to differentiate it from the original and in practice is only an improvement of the metric proposed by Albrecht. Similarly, other studies (Hastings, 1995; R.Jeffery and G.Low, 1997; Jones, 1991; Reifer, 1990; Whitmire, 1992; Umholtz and Leitgeb, 1994; Abran et al., 2001) propose improvements on this method and metric originally proposed, trying to solve its drawbacks. Maxwell and Forselius (2000) used an adaptation of function point to measure the productivity of 206 projects contained in a Benchmarking called Experience Database. According to the authors an ideal measure to assess productivity should consider the size, complexity and quality of software. Karner (1993) in his Master’s thesis proposed a method to count the functional size of a systems based on Object-Oriented methodology and concepts. The metric resulting of the proposed method execution is known as UCP (use case points), which is a unit of measurement based on the idea of function points, but uses the concept of use cases as element for the accounting. Based on this work, Arnold and Pedross (1998) used UCP as a out put measure. They do not explicit which unit of measure was used as input. However, we have encountered several studies reporting it uses only for estimation purpose (Anda et al., 2001; Kusumoto et al., 2004) and not for productivity assessment. Moreover, because it is not so popular, there are some proposals which adapts FP measurement to be adopted in environments where use cases, class diagrams and sequence diagrams are used (Antoniol et al., 2003; Cantone et al., 2004). Function Point became one of the most known and used metric to quantify the. 21.

(48) CHAPTER 2. KEY DEVELOPMENTS IN THE FIELD OF SOFTWARE PRODUCTIVITY MEASUREMENT. size of software and consequently the productivity for information systems and real time systems. Nowadays it is largely used in industry with support of world wide associations which maintains and evolve it, as IFPUG4 and NESMA5 . Despite this, it is very criticized and questioned because several assumptions and definitions do not have a theoretical apparatus which validate it, such as the weights assigned to the data functions or transactions functions (DeMarco, 1982; Symons, 1988; Scacchi, 1989; Jones, 1991; Ejiogu, 1991; Fenton, 1994; Fenton and Pfleeger, 1997).. 2.3.3. Design Based Metrics. This section considers the studies which use out put measures based on elements dependent on the structure of the system, and that exists on design conceptual level. The most common examples of metrics in this category is the number of objects and number of modules. Chatman (1995) suggested using a new metric, called Change-Point. According to him, the use of unit size in LOC as output for a productivity metric is inadequate, since LOC size and effort should be independent. Moreover, he states that the counting of size for a program does not predetermines the effort required for its creation (or the reverse). The Change-Point metric is strongly based on the AWI (Abstract Work Item) concept, that according to the author, is related to a need or characteristic of the product being created or modified (examples given by the author were, “update the payroll to reflect the new tax laws” or “improve performance by 5%”). The accounting process was defined by the author as “for each AWI, count 1 for each module modified, created (new), deleted, or affected that is used to vitalize that AWI ”. In another way, this metric takes into account the total units of program created, modified, removed or affected to implement a set of AWIs. Although the authors give examples of how to compute the Change-Points metric, he agrees that both the concept of AWI and program unit are subjective and can vary depending on the person or organization who conduct the counting. This metric was considered as Design based, because the counting process considers the number of system modules. Moser and Nierstrasz (1996) used a new metric called System Meter. This metric is object oriented and is calculated as the sum of all objects size in the system. To calculate the size of an object the internal and external size or complexity are considered. They compared the productivity using function point with system meter. According to the 4 http://www.ifpug.org/ 5 http://www.nesma.nl/. 22.