5.5 Analysis and discussion
5.5.1 Increase effectiveness when compared with text presentation
5.5.1.1 Number of imperfections detected
The CAS example
The group that worked with the CAS example in text (experiment 2, session 2) was able to detect two out of three conflicts but complained that this detection was difficult because the example contained too much ambiguity. The group that worked with the CAS example in jigsaw puzzle (experiment 2, session 3) found and talked about one conflict (not reported) and lots of ambiguities. It is plausible to conclude that the group that worked with text might have found the conflicts because they knew from the previous task they performed with the other system (with jigsaw puzzle) that the purpose was to scan for conflicts.
The main lesson from the CAS example is that it is not adequate (and thus not worthwhile) to present a set of requirements with too much ambiguity using a jigsaw puzzle. It might help to present the requirements in squares/rectangles and give to the text the same treatment that was done for the jigsaw puzzle. In fact, in the performed experiments (not only with CAS system) it happened sometimes that, if there would be a clarification in an ambiguity problem, then conflicts referring the same requirements, were deemed not pertinent to discuss.
189 The Health-Watcher example
With the Health-Watcher system there were significant improvements between the numbers of conflicts detected from experiment 2 to experiment 3. Concerning the jigsaw puzzle presentation from 2 conflicts detected in experiment 2 to 2 conflicts and part of another 2 in experiment 3. Concerning the text, from 1 conflict detected in experiment 2 to all (the 4) conflicts detected in experiment 3. In fact, both the jigsaw puzzle presentation and the text presentation were improved.
The jigsaw puzzle pieces shapes were changed to allow a direct interlocking shape between Performance and Standards, evolving from what is in Figure 3 to Figure 5-6. This enabled the group that worked with the puzzle in Figure 5-6 (experiment 3) to detect the conflict between Performance and Standards, and part of another conflict.
This seems to indicate that the interlocking shapes are important in helping to detect conflicts.
In the text versions the change was to introduce phrases indicating the possible existence of conflicts like: “Detected conflicts between Security and Standards”, with the goal to provide as much as possible the same information in both presentations.
It is interesting to note that for Health-Watcher a slightly better result was achieved in experiment 3 with the text presentation (all the 4 conflicts detected) than with the jigsaw puzzle (2 conflicts detected and part of the other 2). A plausible explanation for the slightly poorer performance of the group working with the jigsaw might be tiredness. In fact this work was performed after this group worked for one hour and a half with another system in text. Anyhow, the group that performed so well with the text presentation, detecting all the four conflicts, were unanimous in preferring to work with the jigsaw puzzle.
190 The CMS example
With the CMS example, there was much better performance, in the number of conflicts detected, with the jigsaw puzzle presentation compared to the text presentation. In fact with the text presentation, in one session (experiment 2) no conflict was detected and in the other (experiment 3) just part of one conflict was detected. In this case the fact that the text presentation was improved for experiment 3 with the phrases indicating the presence of conflicts seems to have had no positive effect (or a marginal one).
With the jigsaw puzzle presentation in experiment 1 every conflict was detected (from the three conflicts with visual cue). In experiment 2 the participants detected one conflict and part of two other conflicts (there were three conflicts with visual cue). In experiment 3 the participants detected two conflicts and part of a third one (there were four conflicts with visual cue).
5.5.1.2 Preferences between jigsaw puzzle and text presentations
Concerning the preference of performing the identification and analysis of conflicts and ambiguities, with text or with the jigsaw puzzle presentation, all the 22 participants except one (who showed no preference), preferred to work with the jigsaw puzzle. A number of participants could explain why they definitely preferred the jigsaw puzzle presentation. The main reasons are described in Section 5.4.4.1.
Interestingly, some participants could also perceive the danger that the jigsaw puzzle presentation brings: if the person that treats information is biased then participants in
191 the meeting would also be biased. This is not a danger in itself but indeed a responsibility of the requirements engineers in charge.
The two groups that worked with the CAS example (with whatever presentation) believed that with this example the comparison was unfair, because the CAS example had too much ambiguity. They provided their opinion while making an effort to abstract from this problem. The improvement of text with the phrases indicating the presence of conflicts did not influence the result, since in experiment 3 (the one where the text was improved) the preference for the jigsaw puzzle was unanimous.
One participant reported having no preference for either presentation. Another preferred text, but when interviewed for the survey on the picture background, wished to change his opinion saying “the puzzle could provide an interesting tool in an interactive table”. A third participant initially thought the jigsaw puzzle would be useful for analysts only, in a more technical-oriented tool “with the puzzle inside it”. This third participant, when interviewed for the survey on the picture background, changed his position providing suggestions on how the jigsaw could be used with customers, for instance using the colours of the company in the picture background.
5.5.1.3 Conclusion
One of the hypothesis evaluated in this research work, was that the jigsaw puzzle metaphor and the method to work in stakeholders group consultation meetings, increases effectiveness in the communication and handling of ambiguity and conflict in requirements, compared with using text presentation. The experiments’ results demonstrate that both the jigsaw puzzle metaphor and the method exhibit a very
192 good potential to increase their effectiveness in communicating and handling ambiguity and conflict in stakeholders meetings, when compared to what is the common mode of performing this task in text presentation.
The most significant result sustaining this statement is that all participants (17 in experiments 2 and 3), except one (who showed no preference), preferred to work with the jigsaw puzzle than with the text presentation. In experiment 1 (where 5 of the participants were knowledgeable in requirements engineering and thus able to evaluate the comparison of the proposed approach against text) the participants concluded: “the jigsaw puzzle-based presentation is really good in helping identifying problems and conflicts”. It is also significant that a number of participants are aware of and explain the jigsaw puzzle features that make them prefer to work with it.
The examples studied were few (thus with no statistical significance). However, both in the CMS example and the Health-Watcher, the number of conflicts detected with the jigsaw puzzle was, in general, greater than the number detected in the textual presentation. There is only a slightly better result with the text presentation for Health-Watcher in experiment 3. This might be explained by tiredness as was sustained in the analysis of Health-Watcher example (Section 5.5.1.1). It is relevant to note that, in this experiment 3, the textual presentation was not in normal plain text, but text improved with the phrases saying “Detected imperfections between concern X and concern Y”.
The number of conflicts and ambiguities well communicated and thus identified, as well as, the suggestions for handling the imperfections can improve even more significantly, with an interactive digital support. This is the researchers’ belief, together with the (enthusiastic) positive opinion of the participants towards having the jigsaw puzzle metaphor digitally and interactively supported.
193 In itself just the fact that there is a digital support (other than pure text) for analysis and rational descriptions (e.g. comments, arrows relating requirements) can bring enormous benefits: these analysis and rationale are not lost (are recorded), and can easily be distributed.
If the digital support also provides intelligent real-time interaction, enabling that the user changes, for instance, the text of a requirement and the interlocking shapes are redesigned according to a recalculation of possibility of conflicts, then we can say the (digitally supported) tool is enabling us to move towards minimisation of conflicts.
Furthermore, if this progress in work is automatically recorded, its effects are visually observable by the users involved and can be discussed “now”, instead of in the next meeting, after the ones involved “calculate” the possible side-effects. This would represent a huge improvement in handling of requirements conflicts.
However, the enormous potential foreseen for a digitally supported jigsaw puzzle can only be effective if this tool is well designed, using the best practices of usability, and in particular respecting the characteristics of RE task and its users.
194