Story reading by novices

We then ran a user feedback session to evaluate the prototype from a reader’s perspective, and thus close the story communication cycle. We conducted private sessions that lasted between 40 and 50 minutes, with 5 BI novices. All were IT professionals, knew what a dashboard is but had never worked with one. Two had heard of BI reports but never used one.

Participants were asked to (1) Read a BI report created by one of our experts, (2) Read a BI story (created by an expert report creator in the previous session, and (3) Explore our prototype.

The story presented the progress of a system development project, from different perspec- tives: a General (how are the projects evolving in terms of finished code in a sprints time line, how many code components are added to a waiting list, and how many are done each week with a categorization of components as critical, major or minor), and a Detailed perspective (the progress of each development team in coding and testing each code com- ponent). In the story the development is not progressing according to plan because many bugs fixed are not critical, whereas new bug reports coming in add critical bugs to the

# Question

Q1 Why is the trend line of ”user story” in the pink line chart stable in the first spring?

Q2 Why is there no progress in the project ”user story” in the pink chart?

Q3 What is the difference between the two yellow line charts?

Q4 What is the number of critical items done last week?

Q5 Why are the critical non developed items increasing?

Q6 Is the "User Story" project meeting the expectations? Why?

Table 5.1: Users were asked this group of comprehension question related to the content of the story and charts, after reading first the report then the story.

waiting list. On chart in the bottom right highlighted the problem and the rest provide details.

5.7.2.1 Reading the report

When given the report, all BI novice readers read the first page dashboard from left to right and up to down (which in fact is not the ”author” suggested order). They characterized the report as a presentation of facts, that they understood the goal (progress of a project) and what the charts displayed (e.g. bugs in waiting list) with the aid of titles and legends on each chart. But they all struggled to finding what was the problem illustrated in the report. Only one participant figured out that the project development is not progressing over time which is not normal, nevertheless she could not understand why.

This clearly demonstrated our experts’ comment that reports cannot be read without the supporting material.

5.7.2.2 Reading the story

Participants found that reading a story was easier ”the story showed the facts in an un- derstandable manner”. The 5 readers found the system easy to use, understood the story, and were able to answer correctly several comprehension question related to the content of the story and charts Table5.1.

We report here our main observations and readers’ comments:

• Persistence: All users tended to remember even detailed aspects of the story. They only went back to the system when answering detailed quantitative questions (e.g.

to retrieve numbers from charts), while all general comprehension questions were answered without looking back to the story visualizations and comments. They also tended to answer using the same terms used by the expert analyst in their comments.

When asked similar questions after reading just the report, they went back to the report each time to search for answers.

• Confidence: Readers were not confident about their understanding of the report since they had to draw their own conclusions. They expressed worry that they may have misunderstand or not noticed some important points.While all readers felt confident when reading the story, specially since their interpretations are confirmed by another person (analyst comments).

• Guidance: all readers appreciated the guidance in reading the story both static and playback. Participants commented on the importance of both modes: ”the static mode permits to understand the whole story”, while playback ”is easier to follow”

the story sequence. Particularly for the playback, they commented on how it helped them focus and prevented them from spending time on unimportant data. Four participants preferred the fade mode over the default highlight mode, and 3 preferred to add the max mode when focusing on an entity.

• Understanding: all users understand the entire story using our tool, and found it easy to read and interpret. They also commented how it was easier to find answers both to qualitative and quantitative questions. The story structure showed clearly what is the problem, how to analyze it and how to find the cause. On the other hand they described the report as ambiguous, as they couldn’t find the relation between charts or KPIs. When reading the simple report, each reader can choose his own way in structuring and analyzing the data and spend time in exploring in detail each chart to find relations. In the end most could just deduce that all charts represent the same data but from different perspectives.

• Transmitting knowledge: All readers found the annotations very helpful in explaining the relationships between charts and KPIs, and in teaching them the analysis logic they should follow. Four mentioned that the system can ”aid in transmitting different knowledge in the company between different users”, and 2 participants said they would like to be able to use the system to communicate with their team their own data (even if they are not analysts).

• Engagement: Three readers out of 5 got very engaged with the story and began searching in other detailed charts to find how this problem can be solved. As one mentioned ”stories are more encouraging than static reports”.

The comparison of BI reports without supplementary material over our story prototype is clearly unfair and our goal is not to prove its superiority in terms of communication value. It mainly serves to illustrate how our stand-alone prototype could be an effective means of communicating BI stories without additional material. The comparison of our tool against the collective BI story material used today in terms of comprehension is future work, but our experts already identified the gains it provides in terms of saving time on report creation, and allowing the sharing of interactive stories.