Predictive aids can lead to sustained attention decrements in the detection of non-routine critical events in event monitoring

Dev Minotra, Michael D. McNeese

Research output: Contribution to journalArticle

5 Citations (Scopus)

Abstract

Task performance in uncertain and complex environments is dependent on human effectiveness in dynamic prioritisation and attention allocation. Non-routine critical events place a high demand on cognitive resources in operators, and they require operators to dynamically re-prioritise sub-tasks calling upon the need for predictive aids. Predictive aids provide real-time prediction of system variables, and they can potentially facilitate the detection of non-routine critical events and dynamic re-prioritisation during such events. However, prior research has reported mixed findings in their effectiveness, and no prior research has been done to test the effectiveness of predictive aids during non-routine critical events that require operators to dynamically prioritise tasks. Our experimental study with 77 participants examined the effect of a predictive aid on prioritisation of non-routine critical events in a cyber security event monitoring task. The predictive aid resulted in decrements in sustained attention as seen in delayed detection of some non-routine critical events and errors in prioritising non-routine critical events. This effect is a result of miscalibration in the importance of looking ahead with the predictive aid. This miscalibration was possibly associated with the look-ahead time, complexity of the predictive aid, and the lack of automation transparency (i.e. mechanism underlying prediction) together affecting its perceived usefulness. The experimental results have the following implications on the design and testing of predictive aids: (1) in a task requiring dynamic prioritisation and detection of non-routine critical events, predictive aids with insufficient look-ahead times can result in decrements in sustained attention; (2) predictive aids should be evaluated with subjective measures of perceived usefulness and workload; (3) predictive aids should be designed to have sufficient look-ahead times and transparency of underlying prediction mechanisms as these would affect visual scanning effort and perceived usefulness; and (4) experimental tests of their effectiveness should involve scenarios long enough that would take sustained attention effects and any potential learning effects into consideration.

Original languageEnglish (US)
Pages (from-to)161-177
Number of pages17
JournalCognition, Technology and Work
Volume19
Issue number1
DOIs
StatePublished - Feb 1 2017

Fingerprint

Transparency
Monitoring
Automation
Scanning
Testing
Prediction
Usefulness
Operator

All Science Journal Classification (ASJC) codes

  • Philosophy
  • Human-Computer Interaction
  • Computer Science Applications

Cite this

@article{3ffa4ac4e13e4d5396dda621fc37121e,
title = "Predictive aids can lead to sustained attention decrements in the detection of non-routine critical events in event monitoring",
abstract = "Task performance in uncertain and complex environments is dependent on human effectiveness in dynamic prioritisation and attention allocation. Non-routine critical events place a high demand on cognitive resources in operators, and they require operators to dynamically re-prioritise sub-tasks calling upon the need for predictive aids. Predictive aids provide real-time prediction of system variables, and they can potentially facilitate the detection of non-routine critical events and dynamic re-prioritisation during such events. However, prior research has reported mixed findings in their effectiveness, and no prior research has been done to test the effectiveness of predictive aids during non-routine critical events that require operators to dynamically prioritise tasks. Our experimental study with 77 participants examined the effect of a predictive aid on prioritisation of non-routine critical events in a cyber security event monitoring task. The predictive aid resulted in decrements in sustained attention as seen in delayed detection of some non-routine critical events and errors in prioritising non-routine critical events. This effect is a result of miscalibration in the importance of looking ahead with the predictive aid. This miscalibration was possibly associated with the look-ahead time, complexity of the predictive aid, and the lack of automation transparency (i.e. mechanism underlying prediction) together affecting its perceived usefulness. The experimental results have the following implications on the design and testing of predictive aids: (1) in a task requiring dynamic prioritisation and detection of non-routine critical events, predictive aids with insufficient look-ahead times can result in decrements in sustained attention; (2) predictive aids should be evaluated with subjective measures of perceived usefulness and workload; (3) predictive aids should be designed to have sufficient look-ahead times and transparency of underlying prediction mechanisms as these would affect visual scanning effort and perceived usefulness; and (4) experimental tests of their effectiveness should involve scenarios long enough that would take sustained attention effects and any potential learning effects into consideration.",
author = "Dev Minotra and McNeese, {Michael D.}",
year = "2017",
month = "2",
day = "1",
doi = "10.1007/s10111-017-0402-x",
language = "English (US)",
volume = "19",
pages = "161--177",
journal = "Cognition, Technology and Work",
issn = "1435-5558",
publisher = "Springer London",
number = "1",

}

Predictive aids can lead to sustained attention decrements in the detection of non-routine critical events in event monitoring. / Minotra, Dev; McNeese, Michael D.

In: Cognition, Technology and Work, Vol. 19, No. 1, 01.02.2017, p. 161-177.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Predictive aids can lead to sustained attention decrements in the detection of non-routine critical events in event monitoring

AU - Minotra, Dev

AU - McNeese, Michael D.

PY - 2017/2/1

Y1 - 2017/2/1

N2 - Task performance in uncertain and complex environments is dependent on human effectiveness in dynamic prioritisation and attention allocation. Non-routine critical events place a high demand on cognitive resources in operators, and they require operators to dynamically re-prioritise sub-tasks calling upon the need for predictive aids. Predictive aids provide real-time prediction of system variables, and they can potentially facilitate the detection of non-routine critical events and dynamic re-prioritisation during such events. However, prior research has reported mixed findings in their effectiveness, and no prior research has been done to test the effectiveness of predictive aids during non-routine critical events that require operators to dynamically prioritise tasks. Our experimental study with 77 participants examined the effect of a predictive aid on prioritisation of non-routine critical events in a cyber security event monitoring task. The predictive aid resulted in decrements in sustained attention as seen in delayed detection of some non-routine critical events and errors in prioritising non-routine critical events. This effect is a result of miscalibration in the importance of looking ahead with the predictive aid. This miscalibration was possibly associated with the look-ahead time, complexity of the predictive aid, and the lack of automation transparency (i.e. mechanism underlying prediction) together affecting its perceived usefulness. The experimental results have the following implications on the design and testing of predictive aids: (1) in a task requiring dynamic prioritisation and detection of non-routine critical events, predictive aids with insufficient look-ahead times can result in decrements in sustained attention; (2) predictive aids should be evaluated with subjective measures of perceived usefulness and workload; (3) predictive aids should be designed to have sufficient look-ahead times and transparency of underlying prediction mechanisms as these would affect visual scanning effort and perceived usefulness; and (4) experimental tests of their effectiveness should involve scenarios long enough that would take sustained attention effects and any potential learning effects into consideration.

AB - Task performance in uncertain and complex environments is dependent on human effectiveness in dynamic prioritisation and attention allocation. Non-routine critical events place a high demand on cognitive resources in operators, and they require operators to dynamically re-prioritise sub-tasks calling upon the need for predictive aids. Predictive aids provide real-time prediction of system variables, and they can potentially facilitate the detection of non-routine critical events and dynamic re-prioritisation during such events. However, prior research has reported mixed findings in their effectiveness, and no prior research has been done to test the effectiveness of predictive aids during non-routine critical events that require operators to dynamically prioritise tasks. Our experimental study with 77 participants examined the effect of a predictive aid on prioritisation of non-routine critical events in a cyber security event monitoring task. The predictive aid resulted in decrements in sustained attention as seen in delayed detection of some non-routine critical events and errors in prioritising non-routine critical events. This effect is a result of miscalibration in the importance of looking ahead with the predictive aid. This miscalibration was possibly associated with the look-ahead time, complexity of the predictive aid, and the lack of automation transparency (i.e. mechanism underlying prediction) together affecting its perceived usefulness. The experimental results have the following implications on the design and testing of predictive aids: (1) in a task requiring dynamic prioritisation and detection of non-routine critical events, predictive aids with insufficient look-ahead times can result in decrements in sustained attention; (2) predictive aids should be evaluated with subjective measures of perceived usefulness and workload; (3) predictive aids should be designed to have sufficient look-ahead times and transparency of underlying prediction mechanisms as these would affect visual scanning effort and perceived usefulness; and (4) experimental tests of their effectiveness should involve scenarios long enough that would take sustained attention effects and any potential learning effects into consideration.

UR - http://www.scopus.com/inward/record.url?scp=85010931291&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85010931291&partnerID=8YFLogxK

U2 - 10.1007/s10111-017-0402-x

DO - 10.1007/s10111-017-0402-x

M3 - Article

AN - SCOPUS:85010931291

VL - 19

SP - 161

EP - 177

JO - Cognition, Technology and Work

JF - Cognition, Technology and Work

SN - 1435-5558

IS - 1

ER -