Text extraction and retrieval from smartphone screenshots: Building a repository for life in media

Agnese Chiatti, Mu Jung Cho, Anupriya Gagneja, Xiao Yang, Miriam Brinberg, Katie Roehrick, Sagnik Ray Choudhury, Nilam Ram, Byron Reeves, C. Lee Giles

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Daily engagement in life experiences is increasingly interwoven with mobile device use. Screen capture at the scale of seconds is being used in behavioral studies and to implement "just-in-time" health interventions. The increasing psychological breadth of digital information will continue to make the actual screens that people view a preferred if not required source of data about life experiences. Effective and efficient Information Extraction and Retrieval from digital screenshots is a crucial prerequisite to successful use of screen data. In this paper, we present the experimental workflow we exploited to: (i) pre-process a unique collection of screen captures, (ii) extract unstructured text embedded in the images, (iii) organize image text and metadata based on a structured schema, (iv) index the resulting document collection, and (v) allow for Image Retrieval through a dedicated vertical search engine application. The adopted procedure integrates different open source libraries for traditional image processing, Optical Character Recognition (OCR), and Image Retrieval. Our aim is to assess whether and how state-of-the-art methodologies can be applied to this novel data set. We show how combining OpenCV-based pre-processing modules with a Long short-term memory (LSTM) based release of Tesseract OCR, without ad hoc training, led to a 74% character-level accuracy of the extracted text. Further, we used the processed repository as baseline for a dedicated Image Retrieval system, for the immediate use and application for behavioral and prevention scientists. We discuss issues of Text Information Extraction and Retrieval that are particular to the screenshot image case and suggest important future work.

Original languageEnglish (US)
Title of host publicationProceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018
PublisherAssociation for Computing Machinery
Pages948-955
Number of pages8
ISBN (Electronic)9781450351911
DOIs
StatePublished - Apr 9 2018
Event33rd Annual ACM Symposium on Applied Computing, SAC 2018 - Pau, France
Duration: Apr 9 2018Apr 13 2018

Publication series

NameProceedings of the ACM Symposium on Applied Computing

Other

Other33rd Annual ACM Symposium on Applied Computing, SAC 2018
CountryFrance
CityPau
Period4/9/184/13/18

Fingerprint

Smartphones
Image retrieval
Optical character recognition
Search engines
Metadata
Mobile devices
Image processing
Health
Processing

All Science Journal Classification (ASJC) codes

  • Software

Cite this

Chiatti, A., Cho, M. J., Gagneja, A., Yang, X., Brinberg, M., Roehrick, K., ... Giles, C. L. (2018). Text extraction and retrieval from smartphone screenshots: Building a repository for life in media. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018 (pp. 948-955). (Proceedings of the ACM Symposium on Applied Computing). Association for Computing Machinery. https://doi.org/10.1145/3167132.3167236
Chiatti, Agnese ; Cho, Mu Jung ; Gagneja, Anupriya ; Yang, Xiao ; Brinberg, Miriam ; Roehrick, Katie ; Choudhury, Sagnik Ray ; Ram, Nilam ; Reeves, Byron ; Giles, C. Lee. / Text extraction and retrieval from smartphone screenshots : Building a repository for life in media. Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018. Association for Computing Machinery, 2018. pp. 948-955 (Proceedings of the ACM Symposium on Applied Computing).
@inproceedings{c84663da580a4f658c5760c8044bb70d,
title = "Text extraction and retrieval from smartphone screenshots: Building a repository for life in media",
abstract = "Daily engagement in life experiences is increasingly interwoven with mobile device use. Screen capture at the scale of seconds is being used in behavioral studies and to implement {"}just-in-time{"} health interventions. The increasing psychological breadth of digital information will continue to make the actual screens that people view a preferred if not required source of data about life experiences. Effective and efficient Information Extraction and Retrieval from digital screenshots is a crucial prerequisite to successful use of screen data. In this paper, we present the experimental workflow we exploited to: (i) pre-process a unique collection of screen captures, (ii) extract unstructured text embedded in the images, (iii) organize image text and metadata based on a structured schema, (iv) index the resulting document collection, and (v) allow for Image Retrieval through a dedicated vertical search engine application. The adopted procedure integrates different open source libraries for traditional image processing, Optical Character Recognition (OCR), and Image Retrieval. Our aim is to assess whether and how state-of-the-art methodologies can be applied to this novel data set. We show how combining OpenCV-based pre-processing modules with a Long short-term memory (LSTM) based release of Tesseract OCR, without ad hoc training, led to a 74{\%} character-level accuracy of the extracted text. Further, we used the processed repository as baseline for a dedicated Image Retrieval system, for the immediate use and application for behavioral and prevention scientists. We discuss issues of Text Information Extraction and Retrieval that are particular to the screenshot image case and suggest important future work.",
author = "Agnese Chiatti and Cho, {Mu Jung} and Anupriya Gagneja and Xiao Yang and Miriam Brinberg and Katie Roehrick and Choudhury, {Sagnik Ray} and Nilam Ram and Byron Reeves and Giles, {C. Lee}",
year = "2018",
month = "4",
day = "9",
doi = "10.1145/3167132.3167236",
language = "English (US)",
series = "Proceedings of the ACM Symposium on Applied Computing",
publisher = "Association for Computing Machinery",
pages = "948--955",
booktitle = "Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018",

}

Chiatti, A, Cho, MJ, Gagneja, A, Yang, X, Brinberg, M, Roehrick, K, Choudhury, SR, Ram, N, Reeves, B & Giles, CL 2018, Text extraction and retrieval from smartphone screenshots: Building a repository for life in media. in Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018. Proceedings of the ACM Symposium on Applied Computing, Association for Computing Machinery, pp. 948-955, 33rd Annual ACM Symposium on Applied Computing, SAC 2018, Pau, France, 4/9/18. https://doi.org/10.1145/3167132.3167236

Text extraction and retrieval from smartphone screenshots : Building a repository for life in media. / Chiatti, Agnese; Cho, Mu Jung; Gagneja, Anupriya; Yang, Xiao; Brinberg, Miriam; Roehrick, Katie; Choudhury, Sagnik Ray; Ram, Nilam; Reeves, Byron; Giles, C. Lee.

Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018. Association for Computing Machinery, 2018. p. 948-955 (Proceedings of the ACM Symposium on Applied Computing).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Text extraction and retrieval from smartphone screenshots

T2 - Building a repository for life in media

AU - Chiatti, Agnese

AU - Cho, Mu Jung

AU - Gagneja, Anupriya

AU - Yang, Xiao

AU - Brinberg, Miriam

AU - Roehrick, Katie

AU - Choudhury, Sagnik Ray

AU - Ram, Nilam

AU - Reeves, Byron

AU - Giles, C. Lee

PY - 2018/4/9

Y1 - 2018/4/9

N2 - Daily engagement in life experiences is increasingly interwoven with mobile device use. Screen capture at the scale of seconds is being used in behavioral studies and to implement "just-in-time" health interventions. The increasing psychological breadth of digital information will continue to make the actual screens that people view a preferred if not required source of data about life experiences. Effective and efficient Information Extraction and Retrieval from digital screenshots is a crucial prerequisite to successful use of screen data. In this paper, we present the experimental workflow we exploited to: (i) pre-process a unique collection of screen captures, (ii) extract unstructured text embedded in the images, (iii) organize image text and metadata based on a structured schema, (iv) index the resulting document collection, and (v) allow for Image Retrieval through a dedicated vertical search engine application. The adopted procedure integrates different open source libraries for traditional image processing, Optical Character Recognition (OCR), and Image Retrieval. Our aim is to assess whether and how state-of-the-art methodologies can be applied to this novel data set. We show how combining OpenCV-based pre-processing modules with a Long short-term memory (LSTM) based release of Tesseract OCR, without ad hoc training, led to a 74% character-level accuracy of the extracted text. Further, we used the processed repository as baseline for a dedicated Image Retrieval system, for the immediate use and application for behavioral and prevention scientists. We discuss issues of Text Information Extraction and Retrieval that are particular to the screenshot image case and suggest important future work.

AB - Daily engagement in life experiences is increasingly interwoven with mobile device use. Screen capture at the scale of seconds is being used in behavioral studies and to implement "just-in-time" health interventions. The increasing psychological breadth of digital information will continue to make the actual screens that people view a preferred if not required source of data about life experiences. Effective and efficient Information Extraction and Retrieval from digital screenshots is a crucial prerequisite to successful use of screen data. In this paper, we present the experimental workflow we exploited to: (i) pre-process a unique collection of screen captures, (ii) extract unstructured text embedded in the images, (iii) organize image text and metadata based on a structured schema, (iv) index the resulting document collection, and (v) allow for Image Retrieval through a dedicated vertical search engine application. The adopted procedure integrates different open source libraries for traditional image processing, Optical Character Recognition (OCR), and Image Retrieval. Our aim is to assess whether and how state-of-the-art methodologies can be applied to this novel data set. We show how combining OpenCV-based pre-processing modules with a Long short-term memory (LSTM) based release of Tesseract OCR, without ad hoc training, led to a 74% character-level accuracy of the extracted text. Further, we used the processed repository as baseline for a dedicated Image Retrieval system, for the immediate use and application for behavioral and prevention scientists. We discuss issues of Text Information Extraction and Retrieval that are particular to the screenshot image case and suggest important future work.

UR - http://www.scopus.com/inward/record.url?scp=85050564120&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85050564120&partnerID=8YFLogxK

U2 - 10.1145/3167132.3167236

DO - 10.1145/3167132.3167236

M3 - Conference contribution

AN - SCOPUS:85050564120

T3 - Proceedings of the ACM Symposium on Applied Computing

SP - 948

EP - 955

BT - Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018

PB - Association for Computing Machinery

ER -

Chiatti A, Cho MJ, Gagneja A, Yang X, Brinberg M, Roehrick K et al. Text extraction and retrieval from smartphone screenshots: Building a repository for life in media. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018. Association for Computing Machinery. 2018. p. 948-955. (Proceedings of the ACM Symposium on Applied Computing). https://doi.org/10.1145/3167132.3167236