Abstract
Mobile phone use is pervasive, yet little is known about task switching on digital platforms and applications. We propose an unobtrusive experience sampling method to observe how individuals use their smartphones by taking screenshots every 5 seconds when the device is on. The purpose of this paper is to incorporate the psychological process into feature extraction, and use these features to effectively predict the task switching behavior on smartphones. Features are extracted from the sequence of screenshots, gauging visual stimulation, cognitive load, velocity and accumulation, sentiment, and time-related factors. Labels of task switching behavior were manually tagged for 87,182 screenshots from 60 subjects. Using random forest, we demonstrate that we can correctly infer a user's task switching behavior from unstructured data in screenshots with up to 77% accuracy, demonstrating it is a viable option to use features of the screenshots to predict task switching behavior.
| Original language | English (US) |
|---|---|
| Title of host publication | CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems |
| Publisher | Association for Computing Machinery |
| ISBN (Electronic) | 9781450359719 |
| DOIs | |
| State | Published - May 2 2019 |
| Event | 2019 CHI Conference on Human Factors in Computing Systems, CHI EA 2019 - Glasgow, United Kingdom Duration: May 4 2019 → May 9 2019 |
Publication series
| Name | Conference on Human Factors in Computing Systems - Proceedings |
|---|
Conference
| Conference | 2019 CHI Conference on Human Factors in Computing Systems, CHI EA 2019 |
|---|---|
| Country | United Kingdom |
| City | Glasgow |
| Period | 5/4/19 → 5/9/19 |
Fingerprint
All Science Journal Classification (ASJC) codes
- Software
- Human-Computer Interaction
- Computer Graphics and Computer-Aided Design
Cite this
}
Using screenshots to predict task switching on smartphones. / Yang, Xiao; Robinson, Thomas; Ram, Nilam; Reeves, Byron.
CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2019. 3313089 (Conference on Human Factors in Computing Systems - Proceedings).Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
TY - GEN
T1 - Using screenshots to predict task switching on smartphones
AU - Yang, Xiao
AU - Robinson, Thomas
AU - Ram, Nilam
AU - Reeves, Byron
PY - 2019/5/2
Y1 - 2019/5/2
N2 - Mobile phone use is pervasive, yet little is known about task switching on digital platforms and applications. We propose an unobtrusive experience sampling method to observe how individuals use their smartphones by taking screenshots every 5 seconds when the device is on. The purpose of this paper is to incorporate the psychological process into feature extraction, and use these features to effectively predict the task switching behavior on smartphones. Features are extracted from the sequence of screenshots, gauging visual stimulation, cognitive load, velocity and accumulation, sentiment, and time-related factors. Labels of task switching behavior were manually tagged for 87,182 screenshots from 60 subjects. Using random forest, we demonstrate that we can correctly infer a user's task switching behavior from unstructured data in screenshots with up to 77% accuracy, demonstrating it is a viable option to use features of the screenshots to predict task switching behavior.
AB - Mobile phone use is pervasive, yet little is known about task switching on digital platforms and applications. We propose an unobtrusive experience sampling method to observe how individuals use their smartphones by taking screenshots every 5 seconds when the device is on. The purpose of this paper is to incorporate the psychological process into feature extraction, and use these features to effectively predict the task switching behavior on smartphones. Features are extracted from the sequence of screenshots, gauging visual stimulation, cognitive load, velocity and accumulation, sentiment, and time-related factors. Labels of task switching behavior were manually tagged for 87,182 screenshots from 60 subjects. Using random forest, we demonstrate that we can correctly infer a user's task switching behavior from unstructured data in screenshots with up to 77% accuracy, demonstrating it is a viable option to use features of the screenshots to predict task switching behavior.
UR - http://www.scopus.com/inward/record.url?scp=85067299569&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85067299569&partnerID=8YFLogxK
U2 - 10.1145/3290607.3313089
DO - 10.1145/3290607.3313089
M3 - Conference contribution
AN - SCOPUS:85067299569
T3 - Conference on Human Factors in Computing Systems - Proceedings
BT - CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems
PB - Association for Computing Machinery
ER -