Multimodal Fission

In Wang et al. (1993) it was observed that redundancy between speech output and text display enabled the user to shorten learning time for use of a graphical interface. The spatial overlay of the different visual modalities like text, graph- ics, or pictures seems to have an impact on the way subjects integrate the chunks of information they receive (M. Hare & Ryan, 1995). Consequently, a multi- modal system should be able to flexibly generate various presentations for one and the same information content in order to meet the individual requirements of users and situations, the resource limitations of the computing system, and so forth. Besides the presentation generation, even the presentation content should be selected according to the user’s information needs, and the determination of which media to use should be made based on the user’s perceptual abilities and preferences.

In a multimodal system, the fission component is the one that chooses the output to be produced on each of the output channels, and then coordinates the output across the channels. Generically speaking, the fission component is responsible for three categories of tasks:

Content selection and structuring - The content to be included in the pre- sentation must be selected and arranged into an overall structure.

Modality selection - The particular output that is to be realized in each of the available modalities must be specified.

Output coordination - The output of each channel should be coordinated with the others so that the resulting output forms a coherent presentation.

Content selection and structuring constitute the task of designing the overall structure of a presentation. In some cases, the content selection and structuring is done by another process or by the user; for example, the content that is to be presented in APT (Mackinlay, 1986) or MAGPIE (Han & Zukerman,1997) is

determined before the fission process begins. However, in other cases, selecting and structuring the content does form part of the fission process. Two approaches for content selection and structuring can be employed in fission components:

schema-based and plan-based.

The notion of a schema was first proposed in McKeown (1985) in the con- text of text generation. A schema encodes a standard pattern of discourse by means of rhetorical predicates that reflect the function each utterance plays in the text. By associating each rhetorical predicate with an access function for an underlying knowledge base, these schemas can be used to guide both the selection of content and its organization into a coherent text to achieve a given commu- nicative goal. Multimodal presentation systems that use schemas to plan their content include COMET (Feiner & McKeown, 1991) and PostGraphe (Fasciano

& Lapalme, 2000).

To overcome the limitations inherent in schema-based approaches, researchers have applied techniques from AI planning research to the problem of constructing discourse plans that explicitly link communicative intentions with communica- tive actions and the information that can be used in their achievement (Moore, 1995). Text planning generally makes use of plan operators – discourse action descriptions that encode knowledge about the ways in which information can be combined to achieve communicative intentions. Extensions of such plan-based approaches have been used in many multimodal presentation systems as well. Ex- amples of such systems include AIMI (Maybury, 1993), FLUIDS (Herzog et al., 1998), MAGIC (Dalal et al., 1996), MAGPIE (Han & Zukerman, 1997), PPP (Andr´e et al., 1998), and AutoBrief (Kerpedjiev et al., 1998). Generally, such a system generalizes communicative acts to multimedia acts and formalizes them as operators in a planning system. To plan a presentation, the system starts with a high-level communicative goal. It then uses hierarchical expansion of plan opera- tors, terminating when all subgoals have been expanded to elementary generation tasks.

The modality selection task can be summed up in the following manner:

“Given a set of data and a set of media, find a media combination that con- veys all data effectively in a given situation” (Andr´e,2000). Before the process of modality selection it is necessary to consider the knowledge that can be used in

that process. This includes the available modalities (Arens et al., 1993;Bernsen, 1997), characterized according to the type of information they can present or the perceptual task they permit, the characteristics of the information to present, the characteristics of the user (Han & Zukerman, 1997), the user task and goals (Maybury, 1993), and the resource limitations (Baus et al., 2002; Walker et al., 2002).

Several approaches exist for the modality selection process. With the compo- sition approach the system tries to combine selected primitives or operators, us- ing predefined composition operators (Casner,1991; Fasciano & Lapalme, 2000).

Other systems use rules to allocate the components of the presentation among the modalities (Bateman et al., 2001; Feiner & McKeown, 1991). In the systems that use aplan-based approach to content selection and structuring, modality se- lection takes place as a side effect of selecting among presentation strategies, and the necessary knowledge is encoded in the strategies themselves (Herzog et al., 1998; Maybury, 1993). Other possibility still is the use of a system of competing and cooperative agents to plan the presentations (Han & Zukerman,1997).

Finally, the output coordination task is responsible for ensuring that the com- bined output from the individual generators amounts to a coherent presentation.

The following aspects must be considered, depending on the particular modalities that are used and the emphasis of the presentation system:

Physical layout - When more than one visually-presented modality is used – for example, graphics and text – the individual components of the presentation must be laid out (Feiner & McKeown, 1991; Han & Zukerman, 1997).

Temporal coordination - If the presentation includes dynamic modalities such as speech or animation, these presentation events must be coordinated in time. This is essentially the dynamic analogue of the static physical-layout task, although the requirements and approaches differ somewhat (Andr´e et al., 1998; Dalalet al., 1996).

Referring expressions - Some systems further coordinate their output by pro- ducing multimodal and cross-modal referring expressions – i.e., making ref- erences using multiple modalities or referring to other parts of the presen-

tation (Andr´e et al., 1998; Feiner & McKeown, 1991; Johnson & Rickel, 2000).