Information searching as object of study has been approached in many ways in earlier empirical studies. One approach is to design new kinds of IR systems and methods and their components, and test their performance with automatic runs using a pre-existing test collection. This was presented as the one end of the continuum in Figure 2. This approach is important in the technical development of information systems but it is out of the scope of the present thesis. The difference is analogous to the one Allen (1996, pp. 1-2) made between user-centred and data- centred system design. Readers interested in the traditional IR are referred to
Croft, Metzler and Strohman’s (2010) comprehensive overview of IR methods as well as to Voorhees and Harman’s (2005) work presenting The Text REtrieval Conference, TREC.
A step towards naturalistic approach is adding a user, a searcher, into the study.
Typically, the searcher is given a search task assignment and a time limit, and she has to perform the task by searching for documents using a single information system. The findings may concern the system’s performance with real user, and/or the user’s performance using the system, and/or performing the task.
Aula et al. (2010) used this approach to find out, whether performing a difficult (unsuccessful) search task is visible in the search behaviour. Similar questions were raised by Bell and Ruthven (2004) who used three different complexity levels of search tasks for analysing changes in the participants’ search behaviour. Also Gwizdka and Spence (2006) altered the objective complexity of their search tasks, and analysed whether it affected search behaviour or subjective task complexity.
The above examples presented the idea of task complexity and its effects on search behaviour, but left several questions open. Would the participants have similar search tasks in reality? What about the possible work tasks, or other larger tasks, that bring forth the information needs and the search tasks? If the participants had an option, would they actually search this way, using this tool, and why?
In the narrowest sense, the user test approach is merely a small extension to the experimental setting without a user, if the human part is only a substitute to a script forming queries. Though the benefits of controllability are undisputed, the approach can give little information about task-based information searching.
Borlund (2000) states that real life is typically too noisy to be studied directly;
instead, she proposes using simulated work tasks that – when carefully designed – enable both realism and controllability at the same time. This is of course a desirable aim when studying IIR, since (I)IR inherently includes the aspects of understanding how people search for information in order to support searching better, as well as trying to find technical solutions and designing new applications.
As Blandford and Attfield (2010, p. 4) note, simulated work tasks and related IIR evaluation methods still focus on system performance.
Server-side log studies are realistic in the sense that the queries in the logs are issued in naturalistic situations without any artificial control by the researcher. Also huge amounts of raw data enable powerful statistical analyses. However, this all comes at the cost of contextual knowledge. The data do not carry explicit information about information needs, search tasks, or larger tasks behind the
queries. Broder’s (2002) study combined log analysis and pop-up questionnaire to find out the intentions behind the users’ queries. He ended up stating that no more than every other query is informational – others are navigational or transactional by nature. This may hold true but does not reveal much considering task-based information searching. The questionnaire used got a relatively low response ratio, and it provided somewhat narrow options to choose from with quite far-reaching conclusions. Even if the participants had replied totally accurately, and Broder’s (2002) further log analysis had succeeded in sorting out all search intentions behind each query correctly, we still cannot know what kind of task or situation formed the information need in the first place, and how the queries are related to these diverse situations and tasks.
Self-report methods refer to interviews, diaries and questionnaires; that is, data are representations of searching rather than searching itself. Without a doubt, these methods give valuable information especially about participants’ intentions and thoughts, as these cannot be directly observed. However, participants’ responses as the only data concerning searching may be biased. It is not easy to remember exact actions afterwards, and even if they were directly listed in a diary, for example, they may be rationalised. These views are based on my own experiences in data collection but similar problems have been reported in other studies, as well (e.g.
Kumpulainen et al., 2009). Especially when a participant has difficulties in recalling the searching, she naturally tries to form a coherent story. However, the performance itself may not be that coherent. Information is found in unexpected sources, work suffers from interruptions, people multitask and make mistakes.
These things may affect searching though possibly forgotten to be mentioned, or left unstated since they are “of no interest for the researcher”. It may also be difficult to accurately perceive how often a resource is used, for example.
Some studies combine features of both experimental and naturalistic field studies; these are called controlled field studies in Figure 2. For example, Pharo and Järvelin (2004) studied information searching in the Web in order to develop and test a new analysis method. In an empirical study, described in detail in Pharo (2002), students experienced in Web searching were searching information for their theses and these sessions were recorded. The tasks were authentic but the participants were asked to contact the researcher when they wanted to search the Web so that observation and video recording could be arranged. The data set was rather small and analysed qualitatively. The authenticity was increased by the fact that the students were able to use their own browser bookmarks and even allowed to do teamwork. (Pharo, 2002; Pharo & Järvelin, 2004.) At the time of the study it
was not very common to have a personal computer or access to the Web at home, so the participants would most likely have had to plan the time for searching sessions anyhow.
Vakkari, Pennanen and Serola’s (2003) study of university students’ information searching was conducted in a similar manner. Students writing a research proposal participated in two search sessions, one at the beginning of the proposal process and one at the end. They used one bibliographic database to find relevant references for their work. The researches analysed how the searching was affected by search experience in the studied information system, and how the searching changed from the first session to the second. (Vakkari et al., 2003.)
These two sample studies were heading towards a combination of realism and controllability, but they were somewhat limited in their scope. It would be beneficial to apply similar design to other real-life work tasks in addition to study assignments. Some studies have included the real tasks of students in studies using simulated work tasks (e.g. Borlund, 2000; Li & Hu, 2013). This enables comparing the two types of task, simulated and real, on a general level. However, the tasks the participants bring with them may be more of search task than larger task type depending on the instructions given; and related to their studies or leisure time rather than work outside full-time studying.
Next I will present a few studies that aim at approaching information searching from an authentic viewpoint, in the field. This approach is unique in the sense that it can provide information about how people really search and why (the context).
However, a problem lies in the difficulty of study design and control, since basically anything can happen in the field (cf. Robertson, 2008). A naturalistic approach to task-based information searching requires real-time data collection that includes knowledge about the tasks the participants are performing. The participants’ actions are not controlled in any way. They do the tasks at hand that they need to, and use the information resources available in their work place.
Though the data collection obviously takes time and requires cooperative participants (Sonnenwald et al., 2001), the data are deep and enable discovering also new phenomena.
Garber and Grunes (1992) studied art directors by interviewing them and observing them while they were discussing with customers and searching for suitable photos. The study did not analyse task features per se, but it is interesting because it combined several data collection methods and phases fruitfully and expediently. The study was a continuum of successive research goals:
understanding the work and searching of the participants; forming a model of the
searching process based on the observations; and designing and further testing and developing of a user interface for image searching. (Garber & Grunes, 1992.) This is a good example on how study methods support each other, and how a naturalistic approach is an important part of them.
Kelly (2004) conducted a longitudinal study with a few university students that were handed a laptop for research purposes. The participants searched as they normally would in their everyday life, named their tasks and once a week, in more controlled settings, paired their tasks with documents found during the week (Kelly, 2004). Unfortunately, Kelly (2004) did not analyse the tasks per se but rather how searching of each individual evolved. Thus, these findings are difficult to compare to other studies, especially in the sense of how task features affect information searching. Kellar, Watters and Shepherd (2006a; 2006b) conducted a similar study about students’ Web related tasks. The everyday use of the Web was logged, and the participants reported what tasks they were doing. Kelly’s (2004) and Kellar et al.’s (2006a; 2006b) approach was more likely to introduce leisure-time tasks than work tasks.
Huuskonen and Vakkari (2010) observed the work of social workers. They analysed how a client information system was used to support work tasks.
However, they did not analyse how work task features affected its use.
Hansen (2011) conducted a thorough study about patent engineers’ information searching. He exploited interviews, observation, diaries and logs. Work tasks were typified suitably for the domain, and also task knowledge (named also as task difficulty) was analysed. Work task performance was analysed in terms of information needs, sources, queries and relevance judgments. (Hansen, 2011.)
Kumpulainen (2013) directly observed and logged the searching of researchers of molecular medicine. She categorised the work tasks into complexity classes based on the participants’ prior knowledge about each task, and analysed how it affected the problems encountered, search trails and work task processes.
As presented above, the field of (task-based) information searching and its basic concepts have been of much interest lately. However, realistic studies about work- related searching are still rare. Log studies provide quantitative credibility, whereas smaller-scale user studies or self-report methods provide more exact contextual information that can be connected to search actions. The present thesis provides both qualitative and quantitative data about searching in the context of tasks, and information seeking and (I)IR approaches are combined. Field data are rich and naturalistic. Using several data collection methods provides for a reliable view of
real-life searching. Combining research methods in a single study is not common (Chu, 2015).
3 Research design
The thesis is based on two independent data sets, later referred to as Data Set A and Data Set B. The data collection and analysis methods are partly the same, so they are next discussed together. More detailed information about data collection can be found in the contributed Papers I-VI. Papers I-III are based on Data Set A, Papers IV-VI on Data Set B. Table 1 gives an overview of the two data sets. Data Set A was collected together with my coauthor Sanna Kumpulainen. Data Set B was partly collected in cooperation with Heljä Franssila and Jussi Okkonen (School of Information Sciences, University of Tampere) working in Professor Reijo Savolainen’s Information ergonomics project (Franssila, Okkonen & Savolainen, 2014). In the latter case, it was agreed that no coauthorship will follow since we did not share any analysis phases nor any research questions or interests besides the data.
Table 1. Overview of the data sets.
Data Set A Data Set B
Time of data collection
February 2011 – May 2011 August 2013 - October 2014
Participants 6 22
Organisations 1 6
Work tasks 59 286
Interviews n/a Pre- and post-study
Direct observation
During all analysed work tasks;
several sessions per participant
During one working day per participant
Task
questionnaires
Pre- and post-task Morning and afternoon
Logging n/a Transaction log and screen capture
video Volumes 38 observation sessions, 250 pages of
handwritten notes
77 data collection days, 40200 rows of analysable transaction log
The study is designed to support understanding task-based information searching in the real-life context. The approach is explorative. Another important point is that we wanted to see the actual searching actions; rather conventional methods of only interviewing participants or analysing questionnaire responses did not suffice.
It should be remembered that real-life data do not come as neat and clearly labelled packages to the researcher but the analysis phase requires even detective work.