We chose the focus group before using other evaluation methods, because it would enable us to improve the lenses at a lower cost/risk for software professionals. The research questions for the study were:
• RQ 1 – Are the defense lenses as described in the booklet useful and usable for defending software estimates against pressure during the establishment of commitments?
• RQ 2 – What are the improvements to the defense lenses as described in the booklet?
We selected participants from our network, focusing on covering different perspectives in terms of experience and roles. Thus, we invited people with varied experience with software engineering in terms of years of experience (ranging from one to 23 years) and roles (including people with experience as software developers, testers, and managers).
After selecting participants, we asked them to participate in a role-playing focus group in two different meetings. The first meeting is based on the following steps:
1. Gather all participants in a one-hour meeting (maximum) to explain the study objectives and get their informed consent.
2. Present the “Pressuring forces” and the “Laddering whys” lenses in detail as a starting point to acquaint them with the lenses. Describe the structure of the booklet containing the whole lenses set and give them a copy for further study.
3. Collect data about participants’ personality traits (Big-Five Inventory, with 20-items [228]) and assertiveness levels (Rathus Assertiveness Scale [229], Brazilian version with 28 items), because these can be intervening variables in our study.
The second meeting happened one week after the first. We executed the following procedures:
1. Gathered all participants to resolve any questions they have about the lenses. At this point, collected the data about their doubts.
2. Asked someone to volunteer to represent the side receiving the estimate (a client or a manager) from now on referred to as the receiver. All the other participants played the role of estimator.
3. We randomly chose one scenario from a pool. This scenario represents a real-world situation related to the estimation and the establishment of commitments, where the estimator is expected to apply the lenses. The estimators remained unaware of the chosen scenario. The receiver is expected to conduct the task in the next step according to it.
4. While the receptor studied the chosen scenario and the software specification, the estimators read the specification and defined their estimate. The specification was about a software project entitled SeminarWeb [156], which we chose after searching for detailed specifications used in previous studies about software effort estimation.
5. Considering the chosen scenario and the defense lenses, the estimators and the receiver worked together to agree on a commitment. The estimators presented the estimate, and the receiver applied pressure at the appropriate point, as described in the scenario.
6. After they reached a conclusion, we collected data about the expected outcome from the point of view of the estimators and the receiver. We also asked about the lenses that the estimators used.
7. Executed a final debriefing session to gather overall impressions from the participants regarding the lenses and their improvement suggestions.
We present the scenarios that we created in APPENDIX D. We also included their translation to Brazilian Portuguese in APPENDIX E. For the selected scenario, we expected participants to use the lenses from the Minimal Pack, especially the Assert your Estimate, Pressuring Forces, and Candidate Commitments lenses. The meetings were all remote, so we created and managed videoconference rooms for them as needed.
6.2.1. Data Collection and Analysis
Right after the second meeting, participants answered a questionnaire to collect demographic data. Moreover, an intervening variable in our study is personality because previous literature has shown a connection between personality and conflict resolution styles [230]. For instance, we can expect that people who score lower in openness to experience can be more resistant to accepting to use the defense lenses. Therefore, we collected data on the first questionnaire using the Big-Five Inventory (BFI), with 20-items (BFI-20) [228], which assess individuals regarding five personality traits: extraversion, agreeableness, openness to experience, conscientiousness, and neuroticism. We chose BFI-20 because it is short, a characteristic that makes it more appropriate for industry studies, where participants have little time available. An additional advantageous feature of BFI-20 is that it has been validated for the Brazilian population. We included the BFI-20 in ANNEX A.
Another intervening variable is the individual assertiveness level of the estimator.
Interpersonal assertiveness is the degree to which people speak out and stand for their interests when those interests are not perfectly aligned with others’ [53]. When an estimator respectfully defends their estimate in the face of pressure, they exhibit assertive behavior. When they yield to pressure and change their estimates even when they know this will lead to unrealistic commitments, their behavior is unassertive or passive. Therefore, we considered it important to know more of the general assertiveness characteristics of participants in our study and measure it using a Brazilian adaptation of the Rathus Assertiveness Scale (RAS) [229], composed of 30 items, as we show in ANNEX B.
During the second meeting, data collection happened at three points. The first regards doubts about the lenses, at the meeting onset. We analyzed these data to identify clarifications needed in our artifact. The second point was right after the execution of the simulation with the scenarios. We asked participants to answer each of these questions individually:
• What is your expected result for the scenario regarding whether the estimate is changed or defended?
• Which lenses did you apply? (for estimators only)
• Do you think the lenses contributed to the scenario result? Why (not)?
• Were there specific lenses you did not understand how to apply? Which ones? (for estimators only)
• Were there specific lenses you chose not to apply? Why? (for estimators only)
Next, the third point of data collection was a debriefing session guided by the following questions:
• What do you think we need to change/improve in the lenses or the booklet to make them usable in real scenarios that you face (or faced) when estimating and establishing a commitment?
• Do you have any suggestions to make the lenses more useful for the actual estimating and commitment establishment scenarios you face? Do you have any suggestions to make the lenses more usable (easy to consult and apply) for the actual estimating and commitment establishment scenarios you face?
• Do you have interesting examples (like the ones in the booklet) from your professional lives that can contribute to the lenses or the booklet? If yes, can you share them with us?
• Is there anything else you would like to mention/discuss?
Table 6.1 summarizes all the data we collected and their relationship with the second meeting steps and our focus group’ research questions.
Table 6.1–- Summary of data collection
Step Data RQ
1 Questions raised by participants regarding the lenses 2
5 Issues about the uses of the lenses 1
Step Data RQ 6 The expected result for the scenario (estimate is changed or
defended)
1
6 Reasoning for the expected result 1
6 Lenses applied by the estimator 1
6 Reasoning regarding whether the lenses contributed to the expected result
1
6 Lenses that the estimators did not understand 2
6 Lenses that the estimator chose not to apply 2
7 Overall impression about the negotiation lenses 2
7 Overall improvement suggestions for the lenses 2
The data analysis focused on coding the answers and observations of participants to identify the improvements that we can make to our artifact, using the Grounded Theory procedures [220]. We implemented these improvements as changes to the lenses and to the supplemental material (the examples).