The Evaluation

o aesthetic, privacy and social conventions.

For the experimental and field studies twenty participants; twelve males and eight female, varied from 20 to 53 years, participated in the tests. All had visited the city of Mytilene before and were quite experienced with printed guides and maps in general, but only a few had seen or used a printed tourist guide or a map of Mytilene The participants were all experienced mobile phone users; nine of them had used standalone mobile phone applications in the past (e.g. calendar, address books, games, etc), yet, none had previous experience with electronic tourist guides usage.

For the experimental studies a phone emulator has been used, while for the field studies the participants’ personal mobile phones have been utilized as test tools (after downloading and installing the mobile guide application (see Figure 4-15).

Figure 4-15. A participant navigating with mobile guide in the city of Mytilene.

Figure 4-16. A participant navigating with mobile guide application by entering the number of POI that finds on the 2D map.

Each usability test session comprised four parts, an introduction to the scope of the session and the application, the main testing task, an interview and a questionnaire completion.

During the introduction, the overall project and the application were introduced to the participant, demographic information about them was collected and then the test procedure was briefly introduced. The testing stage included four different tasks which were assigned to each participant and were repeated twice. The four tasks are as follows:

1. Select content items from the web application or the info kiosk and then download the mobile application directly to the mobile device through a mobile network and the Internet;

2. Repeat the first task, but trying to first download the application to a PC and then to the mobile device (through Bluetooth);

3. Access specific content pages via the menu system of the mobile application;

4. Use the application’s 2D map to find the way from a place to another (see Figure 4-15) or use the map from the sign posts to enter the number of POI to visit (see Figure 4-16). It is noted that no GPS- enabled device was used to highlight the participant’s current position on the map.

During experimental studies, participants have been asked to accomplish specific tasks using the mobile application (i.e. it is easy to measure usability attributes and interpret results, while it makes it possible to use video or audio recording to capture participants reaction, including emotions) [29]. Out of the generic usability attributes identified in [63] and [186], usability tests recorded quantitative usability attributes measurements that fit in the nature of our electronic guide application such as:

o effectiveness: the percentage of tasks completed;

o efficiency: time needed to solve tasks in comparison to a pre-defined

‘task completion time goal’;

o learnability: the improvement in task performance in the second trial.

Figure 4-17 summarizes the results of the quantitative usability attributes measurements recorded throughout the usability tests.

98 100 97 77

Task #1 Task #2 Task #3 Task #4 Effectiveness (%)

120 100 80 60 40 20 0

90

50

95

82 100

90 80 70 60 50 40 30 20 10 0

Efficiency (%)

(a) (b)

Learnability (%)

99 90 100 91

Task #1 Task #2 Task #3 Task #4 100

90 70

50 30

0 10 20 40 60 80

(c)

Figure 4-17. Measurement of quantitative usability attributes: (a) effectiveness; (b) efficiency; (c) learnability.

It was found that all participants were able to use the web application to view and select POIs with ease. Also, the 4-step process to build the mobile application was carried out by most of our experimental users without any problems. Table 4-3 shows quantative data which was recorded from the 20 users in regards to average number of POIs selected, average file size of mobile application, average download time via Internet (ADSL) to a PC, average installation times via Bluetooth and average installation times via OTA. The platform was found to be effective in regards to the OTA installation as also the task involving the use scenarios of the mobile guide. Only one participant could not complete the build process because their mobile phone was not Java-compatible, hence not in the list of mobile devices offered for

selection via the web application. This user was guided to use the PDA installation. OTA download was found easier and straight-forward to deal with compared to downloading the application on their desktop first and then to their mobile phone through a Bluetooth connection. In fact, almost half of the participants had never used the Bluetooth connectivity service of their mobile phone. The platform was found to be effective in the use of the 4-step process. Highly effective was the use of the OTA installation (see Figure 4-17). However, most of the participants pointed out that the OTA application download time (~ 1.5 min) was too long and expressed concern of cost incurred due to use of the mobile network. As expected the learnability showed encouraging results due to the fact that users handled the system faster the second time around.

Percentage of users 65% 15% 15% 5%

Average number of POIs selected 24-89 10-23

Standard guide (10 POIs) Average file size of mobile

application

1600-3200 KB

980 KB- 2900 KB

980 KB -1200 KB

Average download time via Internet

(ADSL) to a PC 55 sec 45 sec 45 sec

Average installation times via

Bluetooth 127 sec 110 sec 110 sec

Average installation times via OTA 250 sec 230 sec 230 sec

Used a PDA

Table 4-3. Usability results for mobile application selection, download and installation process.

Each test session ended with an interview (following a verbal protocol) and the completion of a short questionnaire, which revealed a number of qualitative usability attributes (see Figure 4-18) such as: user satisfaction, simplicity, comprehensibility and perceived usefulness. In particular, the

participants rated these attributes in a 0-100 scale (0-24: low, 25-49:

moderate, 50-74: average, 75-100: high).

56%

32%

8%

4%

high average moderate low

44%

28%

20%

8%

high average moderate low

(a) User satisfaction (b) Simplicity of use 48%

20% 28%

4%

high average moderate low

20%

32%

8% 40%

high average moderate low

(d) Comprehensibility (d) Perceived usefulness

Figure 4-18. Measurement of qualitative usability attributes: (a) user satisfaction; (b) simplicity of use; (c) comprehensibility; (d) perceived usefulness.

The experimental and field studies revealed a high degree of satisfaction among participants in terms of the amount and quality of the available tourist information. Most users found the web application simple to use in order locate and select content, to go to the “Download now” section, to generate the personalised mobile application and to install it to their mobile phone. The overall web and mobile platform was found to be highly comprehensible, yet many participants perceived the usage of the platform moderately useful, including the mobile application. On average it was found that users were reluctant in choosing many POIs and relied on the profiling system to decide.

Viewing what the users peers chose gave them an idea of what would be interesting to choose.

Overall, participants appreciated the use of the personal recommendation system as it adapted the application’s content to fit participants’ personal preferences; also, the fact that the application could function on their personal mobile devices brought added a perception of familiarity to them. They also realized the business perspectives of including advertisements in the web and mobile applications. A number of issues/recommendations were raised from participants in the interview and evaluated to suggest future enhancements of the mobile application:

o the ability to view own and other tourists’ comments/ratings (in addition to being able to ‘publish’ tourist comments and ratings for a given sight);

o enhancement of digitized maps so as to provide three zoom levels (instead of two), target highlighting, street names and clickable objects, like city attractions, to provide quick access to important tourist information;

o inclusion of emergency contacts in the city map (pharmacies, hospitals, port & tourist police authorities, etc) and search facilities to locate the shortest contacts, depending on the user’s location;

o provision of daily/weakly weather forecast reports based on the user’s location;

o suggestion of daily, personalized tourist itineraries, dynamically adapted to predicted weather conditions.

Further, this feedback taken from participants will be used to improve the future versions of the platform including application content, enhanced features and services of the mobile application. In fact, some of the evaluators recommendations have been taken into account in our following research steps (see Chapters 5 and 6.