The Seas of Factors that Researchers Explored the Most

The occurrence of unforeseen problems is a factor that impacts estimates negatively.

The occurrence of risks, unexpected events, or technical problems leads to a higher need for effort than estimated and estimation errors [82], [94], [119].

Two of the factors affect all phases. The business area has an impact on the effort [90]

and productivity [91]. The other factor is tool support and availability. Software development tools have an empirical influence over management and testing efforts [162]. Additionally, insufficient tool support for project management is a reason for estimation error [96], and the low availability of required tools leads to higher effort [114].

First, six factors have been investigated employing at least three different research strategies: product size (1 R, 4 F, 4 D), complexity (4 R, 4 F, 1 D), use of historical data (2 R, 5 F, 1 D, 1 L), overall experience (4 R, 2 F, 1 D), team size (1 R, 2 F, 4 D) and turnover (2 R, 2 F, 1 D). These are the most explored factors in our SLM. Most of them were investigated through a combination of research, field, and data strategies—suggesting the generalizability, realism of context, and precision of data regarding the supporting findings. Some of these factors are classic cost drivers, such as product size and complexity, and software companies may not have much control over them. Other factors are more controllable but may not be so easy to implement. Still, software practitioners and organizations can organize themselves to use historical data when estimating, increase their overall experience, regulate team sizes to keep them small, and reduce turnover.

Figure 3.11 - Top factors and the studies' research strategies

All the remaining factors in Figure 3.11 were investigated using two different research strategies. Most of them involved respondents and field strategies, suggesting that the findings supporting such factors are more substantial in generalizability and realism of context.

However, they may be weaker in terms of control over variables and precision of data. Some of these factors indicate that improvements in the requirements engineering process benefit greatly the estimation results, such as reducing changes to requirements or scope (6 R, 6 F), working on getting clear requirements specifications (6 R, 4 F), and minimizing misunderstanding of requirements (2R, 4 F). Other factors are related to improvements to the estimation process, such as padding (3 R, 3 F), using standards in estimation (3 R, 3 F), spending enough effort and resources on estimation (4 R, 2 F), and anticipating project’ participant skills (2 R, 3 F). Unexpected events also have their role: overlooked and unplanned tasks (3 R, 4 F). Reducing such events—possibly with the use of checklists, another factor from SEXTAMT—is necessary.

However, improving the requirements engineering and the estimation processes might prove insufficient. Political issues such as pressure (5 R, 3 F) have an impact on estimation results. The management process is an additional concern: adequate monitoring and control (3 R, 2 F) are necessary to ensure fewer errors. Another piece in the puzzle is related to experience: familiarity with the technology (3 R, 2 F). Finally, team issues also have their role, through the team communication and collaboration factor (4 R, 1 F).

A few factors presented in Figure 3.11 involved the combination of field and laboratory strategies, suggesting their strength regarding realism and the control over variables. The first reinforces the importance of the estimation process: the combination strategy of individual estimates (3 F, 3 L). The existing evidence favor group estimation over averaging the individual estimates. The second factor pertains to the realm of biases affecting estimates: anchoring (1 F, 4 L). The third one relates to skill issues: technical skill (3 F, 2 L).

Many of the top factors were probably investigated extensively because of their true impact on the estimates. A few of the other factors may have been investigated because of a controversial result. For instance, despite most of the results favoring group estimation over averaging regarding the combination strategy of individual estimates, one study found the opposite when participants were students [106].

The SEXTAMT factors excluded from Figure 3.11 were reported in four or fewer papers and investigated through no more than two research strategies. They can further enrich our understanding of the impact of the requirements and the estimation process, for instance. They

expand our perspectives to other directions as well, such as the impact of product characteristics, client and user issues, environment, attitudes and maturity, and testing and rework.

In any case, software organizations and practitioners aiming to diagnose the factors more relevant to their context to improve their estimation results can use the SEXTAMT factors to guide what to include in internal surveys, for instance. Practitioners can also use the SEXTAMT factors (especially those classified as value adjusting characteristics) to build internal checklists. For instance, Usman et al. [111] proposed a process to build checklists to support expert judgment estimation, and the first step is to understand the estimation context.

This step has the objective to elicit the factors that should be included in the checklists, either by surveying the literature on the search for effort or cost drivers (top-down approach) and/or the software experts involved in the estimation process (bottom-up approach). Therefore, we support their checklist build process by giving practitioners a map of factors for the top-down approach in this first step, saving them from surveying the literature themselves—a process that involves high costs.

In addition, some of the SEXTAMT factors can be helpful in the debiasing and decision hygiene strategy that Kahneman, Sibony, and Sunstein [208] proposed to help improve judgments in general. Their strategy involves decision observers: people in charge of observing others making judgments in real-time to identify whether biases are pushing them away from the best possible judgment. Decision observers use checklists to accomplish their tasks.

Kahneman, Sibony, and Sunstein [208] suggest that practitioners adjust these checklists to their specific domain. The SEXTAMT factors can guide such adaptation to the software estimation domain. Particularly, the factors from the bias and the estimation process seas at SEXTAMT can provide valuable items in such an adapted checklist.

Also, practitioners can use the SEXTAMT factors as input for risk analysis for their projects, improving their project planning, monitoring, and control. For instance, projects planned to deliver more extensive or more complex products, with less experienced software teams, or where estimators cannot anticipate the participants’ skills when estimating run a larger risk of estimating error and, therefore, of failing to meet their commitments. Thus, project managers of such projects need to be especially caring for monitoring these factors.

Takeaway message 1: There is solid evidence for the factors in the SEXTAMT with 40 of them reported in three or more papers. A few of those—six in total—were investigated by

applying at least three different research strategies. The remaining 29 factors were reported in two studies each, suggesting they can benefit from further investigation.

Takeaway message 2: Practitioners can use the SEXTAMT factors (i) to help diagnose the factors more relevant to change in their contexts, in software process improvement initiatives; (ii) to build supporting checklists for their estimation activities when using expert judgment; (iii) to improve their estimation results in real-time as part of debiasing interventions; or (iv) as input to risk analysis of software projects.

Takeaway message 3: Practitioners interested in improving their estimation can rely on the existing evidence that points to the need for improving the requirements engineering and the estimation process, but also indicates the necessity of considering factors associated with political issues, the management process, experience, team issues, biases, and technical skills.