TY - GEN
T1 - Text extraction from smartphone screenshots to archive in situ Media Behavior
AU - Chiatti, Agnese
AU - Yang, Xiao
AU - Brinberg, Miriam
AU - Cho, Mu Jung
AU - Gagneja, Anupriya
AU - Ram, Nilam
AU - Reeves, Byron
AU - Giles, C. Lee
N1 - Funding Information:
The authors would like to acknowledge the Center for Advanced Study of Behavioral Science at Stanford University where the project was initially developed and the direct funders of this project: the Stanford University Cyber Social Initiative, the Knight Foundation, and the Pennsylvania State University Colleges of Informations Sciences & Technology and Health & Human Development.
Publisher Copyright:
© 2017 Copyright held by the owner/author(s).
PY - 2017/12/4
Y1 - 2017/12/4
N2 - Life experiences are increasingly intertwined with digital devices, suggesting screens as a preferred, if not required, data source for behavioral studies and health interventions. Text Information Extraction from digital screenshots is then a key prerequisite to the overall accuracy of analyses regarding media behaviors. This unique image data set offers the opportunity i) to test existing Image Processing and Text Recognition methods, and ii) to identify and discuss the computational challenges specific to the considered case. Our aim is to assess whether and how state-of-the-art methodologies can be applied to this novel data set. We show how combining OpenCV-based pre-processing with a Long short-term memory (LSTM) based release of Tesseract OCR, without ad hoc training, ensured a 74% text accuracy at the character level. The implications and incidence of different error factors on the resulting quality of text are discussed, prompting the discussion of future research trajectories.
AB - Life experiences are increasingly intertwined with digital devices, suggesting screens as a preferred, if not required, data source for behavioral studies and health interventions. Text Information Extraction from digital screenshots is then a key prerequisite to the overall accuracy of analyses regarding media behaviors. This unique image data set offers the opportunity i) to test existing Image Processing and Text Recognition methods, and ii) to identify and discuss the computational challenges specific to the considered case. Our aim is to assess whether and how state-of-the-art methodologies can be applied to this novel data set. We show how combining OpenCV-based pre-processing with a Long short-term memory (LSTM) based release of Tesseract OCR, without ad hoc training, ensured a 74% text accuracy at the character level. The implications and incidence of different error factors on the resulting quality of text are discussed, prompting the discussion of future research trajectories.
UR - http://www.scopus.com/inward/record.url?scp=85040585450&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85040585450&partnerID=8YFLogxK
U2 - 10.1145/3148011.3154468
DO - 10.1145/3148011.3154468
M3 - Conference contribution
AN - SCOPUS:85040585450
T3 - Proceedings of the Knowledge Capture Conference, K-CAP 2017
BT - Proceedings of the Knowledge Capture Conference, K-CAP 2017
PB - Association for Computing Machinery, Inc
T2 - 9th International Conference on Knowledge Capture, K-CAP 2017
Y2 - 4 December 2017 through 6 December 2017
ER -