Speech-gesture driven multimodal interfaces for Crisis Management

Rajeev Sharma, Mohammed Yeasin, Nils Krahnstoever, Ingmar Rauschert, Guoray Cai, Isaac Brewer, Alan M. MacEachren, Kuntal Sengupta

Research output: Contribution to journalArticle

76 Citations (Scopus)

Abstract

Emergency response requires strategic assessment of risks, decisions, and communications that are time critical while requiring teams of individuals to have fast access to large volumes of complex information and technologies that enable tightly coordinated work. The access to this information by crisis management teams in emergency operations centers can be facilitated through various human-computer interfaces. Unfortunately, these interfaces are hard to use, require extensive training, and often impede rather than support teamwork. Dialogue-enabled devices, based on natural, multimodal interfaces, have the potential of making a variety of information technology tools accessible during crisis management. This paper establishes the importance of multimodal interfaces in various aspects of crisis management and explores many issues in realizing successful speech-gesture driven, dialogue-enabled interfaces for crisis management. This paper is organized in five parts. The first part discusses the needs of crisis management that can be potentially met by the development of appropriate interfaces. The second part discusses the issues related to the design and development of multimodal interfaces in the context of crisis management. The third part discusses the state of the art in both the theories and practices involving these human-computer interfaces. In particular, it describes the evolution and implementation details of two representative systems, Crisis Management (XISM) and Dialog Assisted Visual Environment for Geoinformation (DAVE_G). The fourth part speculates on the short-term and long-term research directions that will help addressing the outstanding challenges in interfaces that support dialogue and collaboration. Finally, the fifth part concludes the paper.

Original languageEnglish (US)
Pages (from-to)1327-1353
Number of pages27
JournalProceedings of the IEEE
Volume91
Issue number9
DOIs
StatePublished - Sep 1 2003

Fingerprint

Interfaces (computer)
Information technology
Communication

All Science Journal Classification (ASJC) codes

  • Electrical and Electronic Engineering

Cite this

Sharma, R., Yeasin, M., Krahnstoever, N., Rauschert, I., Cai, G., Brewer, I., ... Sengupta, K. (2003). Speech-gesture driven multimodal interfaces for Crisis Management. Proceedings of the IEEE, 91(9), 1327-1353. https://doi.org/10.1109/JPROC.2003.817145
Sharma, Rajeev ; Yeasin, Mohammed ; Krahnstoever, Nils ; Rauschert, Ingmar ; Cai, Guoray ; Brewer, Isaac ; MacEachren, Alan M. ; Sengupta, Kuntal. / Speech-gesture driven multimodal interfaces for Crisis Management. In: Proceedings of the IEEE. 2003 ; Vol. 91, No. 9. pp. 1327-1353.
@article{87f56ae7745b48e5a72d863d24d00081,
title = "Speech-gesture driven multimodal interfaces for Crisis Management",
abstract = "Emergency response requires strategic assessment of risks, decisions, and communications that are time critical while requiring teams of individuals to have fast access to large volumes of complex information and technologies that enable tightly coordinated work. The access to this information by crisis management teams in emergency operations centers can be facilitated through various human-computer interfaces. Unfortunately, these interfaces are hard to use, require extensive training, and often impede rather than support teamwork. Dialogue-enabled devices, based on natural, multimodal interfaces, have the potential of making a variety of information technology tools accessible during crisis management. This paper establishes the importance of multimodal interfaces in various aspects of crisis management and explores many issues in realizing successful speech-gesture driven, dialogue-enabled interfaces for crisis management. This paper is organized in five parts. The first part discusses the needs of crisis management that can be potentially met by the development of appropriate interfaces. The second part discusses the issues related to the design and development of multimodal interfaces in the context of crisis management. The third part discusses the state of the art in both the theories and practices involving these human-computer interfaces. In particular, it describes the evolution and implementation details of two representative systems, Crisis Management (XISM) and Dialog Assisted Visual Environment for Geoinformation (DAVE_G). The fourth part speculates on the short-term and long-term research directions that will help addressing the outstanding challenges in interfaces that support dialogue and collaboration. Finally, the fifth part concludes the paper.",
author = "Rajeev Sharma and Mohammed Yeasin and Nils Krahnstoever and Ingmar Rauschert and Guoray Cai and Isaac Brewer and MacEachren, {Alan M.} and Kuntal Sengupta",
year = "2003",
month = "9",
day = "1",
doi = "10.1109/JPROC.2003.817145",
language = "English (US)",
volume = "91",
pages = "1327--1353",
journal = "Proceedings of the IEEE",
issn = "0018-9219",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "9",

}

Sharma, R, Yeasin, M, Krahnstoever, N, Rauschert, I, Cai, G, Brewer, I, MacEachren, AM & Sengupta, K 2003, 'Speech-gesture driven multimodal interfaces for Crisis Management', Proceedings of the IEEE, vol. 91, no. 9, pp. 1327-1353. https://doi.org/10.1109/JPROC.2003.817145

Speech-gesture driven multimodal interfaces for Crisis Management. / Sharma, Rajeev; Yeasin, Mohammed; Krahnstoever, Nils; Rauschert, Ingmar; Cai, Guoray; Brewer, Isaac; MacEachren, Alan M.; Sengupta, Kuntal.

In: Proceedings of the IEEE, Vol. 91, No. 9, 01.09.2003, p. 1327-1353.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Speech-gesture driven multimodal interfaces for Crisis Management

AU - Sharma, Rajeev

AU - Yeasin, Mohammed

AU - Krahnstoever, Nils

AU - Rauschert, Ingmar

AU - Cai, Guoray

AU - Brewer, Isaac

AU - MacEachren, Alan M.

AU - Sengupta, Kuntal

PY - 2003/9/1

Y1 - 2003/9/1

N2 - Emergency response requires strategic assessment of risks, decisions, and communications that are time critical while requiring teams of individuals to have fast access to large volumes of complex information and technologies that enable tightly coordinated work. The access to this information by crisis management teams in emergency operations centers can be facilitated through various human-computer interfaces. Unfortunately, these interfaces are hard to use, require extensive training, and often impede rather than support teamwork. Dialogue-enabled devices, based on natural, multimodal interfaces, have the potential of making a variety of information technology tools accessible during crisis management. This paper establishes the importance of multimodal interfaces in various aspects of crisis management and explores many issues in realizing successful speech-gesture driven, dialogue-enabled interfaces for crisis management. This paper is organized in five parts. The first part discusses the needs of crisis management that can be potentially met by the development of appropriate interfaces. The second part discusses the issues related to the design and development of multimodal interfaces in the context of crisis management. The third part discusses the state of the art in both the theories and practices involving these human-computer interfaces. In particular, it describes the evolution and implementation details of two representative systems, Crisis Management (XISM) and Dialog Assisted Visual Environment for Geoinformation (DAVE_G). The fourth part speculates on the short-term and long-term research directions that will help addressing the outstanding challenges in interfaces that support dialogue and collaboration. Finally, the fifth part concludes the paper.

AB - Emergency response requires strategic assessment of risks, decisions, and communications that are time critical while requiring teams of individuals to have fast access to large volumes of complex information and technologies that enable tightly coordinated work. The access to this information by crisis management teams in emergency operations centers can be facilitated through various human-computer interfaces. Unfortunately, these interfaces are hard to use, require extensive training, and often impede rather than support teamwork. Dialogue-enabled devices, based on natural, multimodal interfaces, have the potential of making a variety of information technology tools accessible during crisis management. This paper establishes the importance of multimodal interfaces in various aspects of crisis management and explores many issues in realizing successful speech-gesture driven, dialogue-enabled interfaces for crisis management. This paper is organized in five parts. The first part discusses the needs of crisis management that can be potentially met by the development of appropriate interfaces. The second part discusses the issues related to the design and development of multimodal interfaces in the context of crisis management. The third part discusses the state of the art in both the theories and practices involving these human-computer interfaces. In particular, it describes the evolution and implementation details of two representative systems, Crisis Management (XISM) and Dialog Assisted Visual Environment for Geoinformation (DAVE_G). The fourth part speculates on the short-term and long-term research directions that will help addressing the outstanding challenges in interfaces that support dialogue and collaboration. Finally, the fifth part concludes the paper.

UR - http://www.scopus.com/inward/record.url?scp=0742311416&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0742311416&partnerID=8YFLogxK

U2 - 10.1109/JPROC.2003.817145

DO - 10.1109/JPROC.2003.817145

M3 - Article

VL - 91

SP - 1327

EP - 1353

JO - Proceedings of the IEEE

JF - Proceedings of the IEEE

SN - 0018-9219

IS - 9

ER -

Sharma R, Yeasin M, Krahnstoever N, Rauschert I, Cai G, Brewer I et al. Speech-gesture driven multimodal interfaces for Crisis Management. Proceedings of the IEEE. 2003 Sep 1;91(9):1327-1353. https://doi.org/10.1109/JPROC.2003.817145