Using screenshots to predict task switching on smartphones

Xiao Yang; Thomas Robinson; Nilam Ram; Byron Reeves

doi:10.1145/3290607.3313089

Using screenshots to predict task switching on smartphones

Xiao Yang, Thomas Robinson, Nilam Ram, Byron Reeves

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Mobile phone use is pervasive, yet little is known about task switching on digital platforms and applications. We propose an unobtrusive experience sampling method to observe how individuals use their smartphones by taking screenshots every 5 seconds when the device is on. The purpose of this paper is to incorporate the psychological process into feature extraction, and use these features to effectively predict the task switching behavior on smartphones. Features are extracted from the sequence of screenshots, gauging visual stimulation, cognitive load, velocity and accumulation, sentiment, and time-related factors. Labels of task switching behavior were manually tagged for 87,182 screenshots from 60 subjects. Using random forest, we demonstrate that we can correctly infer a user's task switching behavior from unstructured data in screenshots with up to 77% accuracy, demonstrating it is a viable option to use features of the screenshots to predict task switching behavior.

Original language	English (US)
Title of host publication	CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems
Publisher	Association for Computing Machinery
ISBN (Electronic)	9781450359719
DOIs	https://doi.org/10.1145/3290607.3313089
State	Published - May 2 2019
Event	2019 CHI Conference on Human Factors in Computing Systems, CHI EA 2019 - Glasgow, United Kingdom Duration: May 4 2019 → May 9 2019

Publication series

Name	Conference on Human Factors in Computing Systems - Proceedings

Conference

Conference	2019 CHI Conference on Human Factors in Computing Systems, CHI EA 2019
Country	United Kingdom
City	Glasgow
Period	5/4/19 → 5/9/19

All Science Journal Classification (ASJC) codes

Software
Human-Computer Interaction
Computer Graphics and Computer-Aided Design

Access to Document

10.1145/3290607.3313089

Fingerprint Dive into the research topics of 'Using screenshots to predict task switching on smartphones'. Together they form a unique fingerprint.

Cite this

@inproceedings{80b82419aa684143b85e46982f0bad93,

title = "Using screenshots to predict task switching on smartphones",

abstract = "Mobile phone use is pervasive, yet little is known about task switching on digital platforms and applications. We propose an unobtrusive experience sampling method to observe how individuals use their smartphones by taking screenshots every 5 seconds when the device is on. The purpose of this paper is to incorporate the psychological process into feature extraction, and use these features to effectively predict the task switching behavior on smartphones. Features are extracted from the sequence of screenshots, gauging visual stimulation, cognitive load, velocity and accumulation, sentiment, and time-related factors. Labels of task switching behavior were manually tagged for 87,182 screenshots from 60 subjects. Using random forest, we demonstrate that we can correctly infer a user's task switching behavior from unstructured data in screenshots with up to 77% accuracy, demonstrating it is a viable option to use features of the screenshots to predict task switching behavior.",

author = "Xiao Yang and Thomas Robinson and Nilam Ram and Byron Reeves",

note = "Publisher Copyright: {\textcopyright} 2019 Copyright held by the owner/author(s). Copyright: Copyright 2019 Elsevier B.V., All rights reserved.; 2019 CHI Conference on Human Factors in Computing Systems, CHI EA 2019 ; Conference date: 04-05-2019 Through 09-05-2019",

year = "2019",

month = may,

day = "2",

doi = "10.1145/3290607.3313089",

language = "English (US)",

series = "Conference on Human Factors in Computing Systems - Proceedings",

publisher = "Association for Computing Machinery",

booktitle = "CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems",

}

Yang, X, Robinson, T, Ram, N & Reeves, B 2019, Using screenshots to predict task switching on smartphones. in CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems., 3313089, Conference on Human Factors in Computing Systems - Proceedings, Association for Computing Machinery, 2019 CHI Conference on Human Factors in Computing Systems, CHI EA 2019, Glasgow, United Kingdom, 5/4/19. https://doi.org/10.1145/3290607.3313089

Using screenshots to predict task switching on smartphones. / Yang, Xiao; Robinson, Thomas; Ram, Nilam; Reeves, Byron.

CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2019. 3313089 (Conference on Human Factors in Computing Systems - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Using screenshots to predict task switching on smartphones

AU - Yang, Xiao

AU - Robinson, Thomas

AU - Ram, Nilam

AU - Reeves, Byron

PY - 2019/5/2

Y1 - 2019/5/2

N2 - Mobile phone use is pervasive, yet little is known about task switching on digital platforms and applications. We propose an unobtrusive experience sampling method to observe how individuals use their smartphones by taking screenshots every 5 seconds when the device is on. The purpose of this paper is to incorporate the psychological process into feature extraction, and use these features to effectively predict the task switching behavior on smartphones. Features are extracted from the sequence of screenshots, gauging visual stimulation, cognitive load, velocity and accumulation, sentiment, and time-related factors. Labels of task switching behavior were manually tagged for 87,182 screenshots from 60 subjects. Using random forest, we demonstrate that we can correctly infer a user's task switching behavior from unstructured data in screenshots with up to 77% accuracy, demonstrating it is a viable option to use features of the screenshots to predict task switching behavior.

AB - Mobile phone use is pervasive, yet little is known about task switching on digital platforms and applications. We propose an unobtrusive experience sampling method to observe how individuals use their smartphones by taking screenshots every 5 seconds when the device is on. The purpose of this paper is to incorporate the psychological process into feature extraction, and use these features to effectively predict the task switching behavior on smartphones. Features are extracted from the sequence of screenshots, gauging visual stimulation, cognitive load, velocity and accumulation, sentiment, and time-related factors. Labels of task switching behavior were manually tagged for 87,182 screenshots from 60 subjects. Using random forest, we demonstrate that we can correctly infer a user's task switching behavior from unstructured data in screenshots with up to 77% accuracy, demonstrating it is a viable option to use features of the screenshots to predict task switching behavior.

UR - http://www.scopus.com/inward/record.url?scp=85067299569&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85067299569&partnerID=8YFLogxK

U2 - 10.1145/3290607.3313089

DO - 10.1145/3290607.3313089

M3 - Conference contribution

AN - SCOPUS:85067299569

T3 - Conference on Human Factors in Computing Systems - Proceedings

BT - CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

PB - Association for Computing Machinery

T2 - 2019 CHI Conference on Human Factors in Computing Systems, CHI EA 2019

Y2 - 4 May 2019 through 9 May 2019

ER -