Abstract
Life experiences are increasingly intertwined with digital devices, suggesting screens as a preferred, if not required, data source for behavioral studies and health interventions. Text Information Extraction from digital screenshots is then a key prerequisite to the overall accuracy of analyses regarding media behaviors. This unique image data set offers the opportunity i) to test existing Image Processing and Text Recognition methods, and ii) to identify and discuss the computational challenges specific to the considered case. Our aim is to assess whether and how state-of-the-art methodologies can be applied to this novel data set. We show how combining OpenCV-based pre-processing with a Long short-term memory (LSTM) based release of Tesseract OCR, without ad hoc training, ensured a 74% text accuracy at the character level. The implications and incidence of different error factors on the resulting quality of text are discussed, prompting the discussion of future research trajectories.
| Original language | English (US) |
|---|---|
| Title of host publication | Proceedings of the Knowledge Capture Conference, K-CAP 2017 |
| Publisher | Association for Computing Machinery, Inc |
| ISBN (Electronic) | 9781450355537 |
| DOIs | |
| State | Published - Dec 4 2017 |
| Event | 9th International Conference on Knowledge Capture, K-CAP 2017 - Austin, United States Duration: Dec 4 2017 → Dec 6 2017 |
Publication series
| Name | Proceedings of the Knowledge Capture Conference, K-CAP 2017 |
|---|
Other
| Other | 9th International Conference on Knowledge Capture, K-CAP 2017 |
|---|---|
| Country | United States |
| City | Austin |
| Period | 12/4/17 → 12/6/17 |
Fingerprint
All Science Journal Classification (ASJC) codes
- Computational Theory and Mathematics
- Software
- Computer Science Applications
- Information Systems
Cite this
}
Text extraction from smartphone screenshots to archive in situ Media Behavior. / Chiatti, Agnese; Yang, Xiao; Brinberg, Miriam; Cho, Mu Jung; Gagneja, Anupriya; Ram, Nilam; Reeves, Byron; Giles, C. Lee.
Proceedings of the Knowledge Capture Conference, K-CAP 2017. Association for Computing Machinery, Inc, 2017. 40 (Proceedings of the Knowledge Capture Conference, K-CAP 2017).Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
TY - GEN
T1 - Text extraction from smartphone screenshots to archive in situ Media Behavior
AU - Chiatti, Agnese
AU - Yang, Xiao
AU - Brinberg, Miriam
AU - Cho, Mu Jung
AU - Gagneja, Anupriya
AU - Ram, Nilam
AU - Reeves, Byron
AU - Giles, C. Lee
PY - 2017/12/4
Y1 - 2017/12/4
N2 - Life experiences are increasingly intertwined with digital devices, suggesting screens as a preferred, if not required, data source for behavioral studies and health interventions. Text Information Extraction from digital screenshots is then a key prerequisite to the overall accuracy of analyses regarding media behaviors. This unique image data set offers the opportunity i) to test existing Image Processing and Text Recognition methods, and ii) to identify and discuss the computational challenges specific to the considered case. Our aim is to assess whether and how state-of-the-art methodologies can be applied to this novel data set. We show how combining OpenCV-based pre-processing with a Long short-term memory (LSTM) based release of Tesseract OCR, without ad hoc training, ensured a 74% text accuracy at the character level. The implications and incidence of different error factors on the resulting quality of text are discussed, prompting the discussion of future research trajectories.
AB - Life experiences are increasingly intertwined with digital devices, suggesting screens as a preferred, if not required, data source for behavioral studies and health interventions. Text Information Extraction from digital screenshots is then a key prerequisite to the overall accuracy of analyses regarding media behaviors. This unique image data set offers the opportunity i) to test existing Image Processing and Text Recognition methods, and ii) to identify and discuss the computational challenges specific to the considered case. Our aim is to assess whether and how state-of-the-art methodologies can be applied to this novel data set. We show how combining OpenCV-based pre-processing with a Long short-term memory (LSTM) based release of Tesseract OCR, without ad hoc training, ensured a 74% text accuracy at the character level. The implications and incidence of different error factors on the resulting quality of text are discussed, prompting the discussion of future research trajectories.
UR - http://www.scopus.com/inward/record.url?scp=85040585450&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85040585450&partnerID=8YFLogxK
U2 - 10.1145/3148011.3154468
DO - 10.1145/3148011.3154468
M3 - Conference contribution
AN - SCOPUS:85040585450
T3 - Proceedings of the Knowledge Capture Conference, K-CAP 2017
BT - Proceedings of the Knowledge Capture Conference, K-CAP 2017
PB - Association for Computing Machinery, Inc
ER -