A reverse turing test for detecting machine-made texts

Jialin Shao, Adaku Uchendu, Dongwon Lee

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

As AI technologies rapidly advance, the artifacts created by machines will become prevalent. As recent incidents by the Deepfake illustrate, then, being able to differentiate man-made vs. machinemade artifacts, especially in social media space, becomes more important. In this preliminary work, in this regard, we formulate such a classification task as the Reverse Turing Test (RTT) and investigate on the contemporary status to be able to classify man-made vs. machine-made texts. Studying real-life machine-made texts in three domains of financial earning reports, research articles, and chatbot dialogues, we found that the classification of man-made vs. machine-made texts can be done at least as accurate as 0.84 in F1 score. We also found some differences between man-made and machine-made in sentiment, readability, and textual features, which can help differentiate them.

Original languageEnglish (US)
Title of host publicationWebSci 2019 - Proceedings of the 11th ACM Conference on Web Science
PublisherAssociation for Computing Machinery, Inc
Pages275-279
Number of pages5
ISBN (Electronic)9781450362023
DOIs
StatePublished - Jun 26 2019
Event11th ACM Conference on Web Science, WebSci 2019 - Boston, United States
Duration: Jun 30 2019Jul 3 2019

Publication series

NameWebSci 2019 - Proceedings of the 11th ACM Conference on Web Science

Conference

Conference11th ACM Conference on Web Science, WebSci 2019
CountryUnited States
CityBoston
Period6/30/197/3/19

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications

Cite this

Shao, J., Uchendu, A., & Lee, D. (2019). A reverse turing test for detecting machine-made texts. In WebSci 2019 - Proceedings of the 11th ACM Conference on Web Science (pp. 275-279). (WebSci 2019 - Proceedings of the 11th ACM Conference on Web Science). Association for Computing Machinery, Inc. https://doi.org/10.1145/3292522.3326042
Shao, Jialin ; Uchendu, Adaku ; Lee, Dongwon. / A reverse turing test for detecting machine-made texts. WebSci 2019 - Proceedings of the 11th ACM Conference on Web Science. Association for Computing Machinery, Inc, 2019. pp. 275-279 (WebSci 2019 - Proceedings of the 11th ACM Conference on Web Science).
@inproceedings{6ef3c318784042e2b5ed775c7d66d4f0,
title = "A reverse turing test for detecting machine-made texts",
abstract = "As AI technologies rapidly advance, the artifacts created by machines will become prevalent. As recent incidents by the Deepfake illustrate, then, being able to differentiate man-made vs. machinemade artifacts, especially in social media space, becomes more important. In this preliminary work, in this regard, we formulate such a classification task as the Reverse Turing Test (RTT) and investigate on the contemporary status to be able to classify man-made vs. machine-made texts. Studying real-life machine-made texts in three domains of financial earning reports, research articles, and chatbot dialogues, we found that the classification of man-made vs. machine-made texts can be done at least as accurate as 0.84 in F1 score. We also found some differences between man-made and machine-made in sentiment, readability, and textual features, which can help differentiate them.",
author = "Jialin Shao and Adaku Uchendu and Dongwon Lee",
year = "2019",
month = "6",
day = "26",
doi = "10.1145/3292522.3326042",
language = "English (US)",
series = "WebSci 2019 - Proceedings of the 11th ACM Conference on Web Science",
publisher = "Association for Computing Machinery, Inc",
pages = "275--279",
booktitle = "WebSci 2019 - Proceedings of the 11th ACM Conference on Web Science",

}

Shao, J, Uchendu, A & Lee, D 2019, A reverse turing test for detecting machine-made texts. in WebSci 2019 - Proceedings of the 11th ACM Conference on Web Science. WebSci 2019 - Proceedings of the 11th ACM Conference on Web Science, Association for Computing Machinery, Inc, pp. 275-279, 11th ACM Conference on Web Science, WebSci 2019, Boston, United States, 6/30/19. https://doi.org/10.1145/3292522.3326042

A reverse turing test for detecting machine-made texts. / Shao, Jialin; Uchendu, Adaku; Lee, Dongwon.

WebSci 2019 - Proceedings of the 11th ACM Conference on Web Science. Association for Computing Machinery, Inc, 2019. p. 275-279 (WebSci 2019 - Proceedings of the 11th ACM Conference on Web Science).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - A reverse turing test for detecting machine-made texts

AU - Shao, Jialin

AU - Uchendu, Adaku

AU - Lee, Dongwon

PY - 2019/6/26

Y1 - 2019/6/26

N2 - As AI technologies rapidly advance, the artifacts created by machines will become prevalent. As recent incidents by the Deepfake illustrate, then, being able to differentiate man-made vs. machinemade artifacts, especially in social media space, becomes more important. In this preliminary work, in this regard, we formulate such a classification task as the Reverse Turing Test (RTT) and investigate on the contemporary status to be able to classify man-made vs. machine-made texts. Studying real-life machine-made texts in three domains of financial earning reports, research articles, and chatbot dialogues, we found that the classification of man-made vs. machine-made texts can be done at least as accurate as 0.84 in F1 score. We also found some differences between man-made and machine-made in sentiment, readability, and textual features, which can help differentiate them.

AB - As AI technologies rapidly advance, the artifacts created by machines will become prevalent. As recent incidents by the Deepfake illustrate, then, being able to differentiate man-made vs. machinemade artifacts, especially in social media space, becomes more important. In this preliminary work, in this regard, we formulate such a classification task as the Reverse Turing Test (RTT) and investigate on the contemporary status to be able to classify man-made vs. machine-made texts. Studying real-life machine-made texts in three domains of financial earning reports, research articles, and chatbot dialogues, we found that the classification of man-made vs. machine-made texts can be done at least as accurate as 0.84 in F1 score. We also found some differences between man-made and machine-made in sentiment, readability, and textual features, which can help differentiate them.

UR - http://www.scopus.com/inward/record.url?scp=85069537165&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85069537165&partnerID=8YFLogxK

U2 - 10.1145/3292522.3326042

DO - 10.1145/3292522.3326042

M3 - Conference contribution

AN - SCOPUS:85069537165

T3 - WebSci 2019 - Proceedings of the 11th ACM Conference on Web Science

SP - 275

EP - 279

BT - WebSci 2019 - Proceedings of the 11th ACM Conference on Web Science

PB - Association for Computing Machinery, Inc

ER -

Shao J, Uchendu A, Lee D. A reverse turing test for detecting machine-made texts. In WebSci 2019 - Proceedings of the 11th ACM Conference on Web Science. Association for Computing Machinery, Inc. 2019. p. 275-279. (WebSci 2019 - Proceedings of the 11th ACM Conference on Web Science). https://doi.org/10.1145/3292522.3326042