Dixit: Interactive visual storytelling via term manipulation

Chao Chun Hsu, Hsin Yu Lin, Yu Hua Chen, Ting Hao Huang, Zi Yuan Chen, Lun Wei Ku

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, we introduce Dixit, an interactive visual storytelling system that the user interacts with iteratively to compose a short story for a photo sequence. The user initiates the process by uploading a sequence of photos. Dixit first extracts text terms from each photo which describe the objects (e.g., boy, bike) or actions (e.g., sleep) in the photo, and then allows the user to add new terms or remove existing terms. Dixit then generates a short story based on these terms. Behind the scenes, Dixit uses an LSTM-based model trained on image caption data and FrameNet to distill terms from each image, and utilizes a transformer decoder to compose a context-coherent story. Users change images or terms iteratively with Dixit to create the most ideal story. Dixit also allows users to manually edit and rate stories. The proposed procedure opens up possibilities for interpretable and controllable visual storytelling, allowing users to understand the story formation rationale and to intervene in the generation process.

Original languageEnglish (US)
Title of host publicationThe Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019
PublisherAssociation for Computing Machinery, Inc
Pages3531-3535
Number of pages5
ISBN (Electronic)9781450366748
DOIs
StatePublished - May 13 2019
Event2019 World Wide Web Conference, WWW 2019 - San Francisco, United States
Duration: May 13 2019May 17 2019

Publication series

NameThe Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019

Conference

Conference2019 World Wide Web Conference, WWW 2019
CountryUnited States
CitySan Francisco
Period5/13/195/17/19

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Software

Fingerprint Dive into the research topics of 'Dixit: Interactive visual storytelling via term manipulation'. Together they form a unique fingerprint.

  • Cite this

    Hsu, C. C., Lin, H. Y., Chen, Y. H., Huang, T. H., Chen, Z. Y., & Ku, L. W. (2019). Dixit: Interactive visual storytelling via term manipulation. In The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019 (pp. 3531-3535). (The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019). Association for Computing Machinery, Inc. https://doi.org/10.1145/3308558.3314131