Online Hate Interpretation Varies by Country, but More by Individual: A Statistical Analysis Using Crowdsourced Ratings

Joni Salminen, Fabio Veronesi, Hind Almerekhi, Soon Gvo Jung, Bernard James Jansen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)

Abstract

Hate is prevalent in online social media. This has resulted in a considerable amount of research in detecting and scoring it. Most computational efforts involve machine learning with crowdsourced ratings as training data. A prominent example of this is the Perspective API., a tool by Google to score toxicity of online comments. However., a major issue in the existing approaches is the lack of consideration for the subjective nature of online hate. While there is research that shows the intensity of hate varies and the hate depends on the context., there is no research that systematically investigates how hate interpretation varies by country or individual. In this exploratory research, we undertake this challenge. We sample crowd workers from 50 countries, have them score the same social media comments for toxicity and then evaluate the differences in the scores., altogether 18.,125 ratings. We find that the interpretation score differences among countries are highly significant. However., the hate interpretations vary more by the individual raters than by countries. These findings suggest that hate scoring systems should consider user-level features when scoring and automating the processing of online hate.

Original languageEnglish (US)
Title of host publication2018 5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages88-94
Number of pages7
ISBN (Electronic)9781538695883
DOIs
StatePublished - Nov 30 2018
Event5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018 - Valencia, Spain
Duration: Oct 15 2018Oct 18 2018

Publication series

Name2018 5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018

Other

Other5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018
CountrySpain
CityValencia
Period10/15/1810/18/18

Fingerprint

hate
statistical analysis
Statistical methods
rating
interpretation
Toxicity
social media
Application programming interfaces (API)
Learning systems
Statistical analysis
Rating
Processing
search engine
Scoring
worker
Social media
lack

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Information Systems and Management
  • Safety, Risk, Reliability and Quality
  • Communication

Cite this

Salminen, J., Veronesi, F., Almerekhi, H., Jung, S. G., & Jansen, B. J. (2018). Online Hate Interpretation Varies by Country, but More by Individual: A Statistical Analysis Using Crowdsourced Ratings. In 2018 5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018 (pp. 88-94). [8554954] (2018 5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SNAMS.2018.8554954
Salminen, Joni ; Veronesi, Fabio ; Almerekhi, Hind ; Jung, Soon Gvo ; Jansen, Bernard James. / Online Hate Interpretation Varies by Country, but More by Individual : A Statistical Analysis Using Crowdsourced Ratings. 2018 5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 88-94 (2018 5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018).
@inproceedings{478a1acc2117451289ca2b1d151ea426,
title = "Online Hate Interpretation Varies by Country, but More by Individual: A Statistical Analysis Using Crowdsourced Ratings",
abstract = "Hate is prevalent in online social media. This has resulted in a considerable amount of research in detecting and scoring it. Most computational efforts involve machine learning with crowdsourced ratings as training data. A prominent example of this is the Perspective API., a tool by Google to score toxicity of online comments. However., a major issue in the existing approaches is the lack of consideration for the subjective nature of online hate. While there is research that shows the intensity of hate varies and the hate depends on the context., there is no research that systematically investigates how hate interpretation varies by country or individual. In this exploratory research, we undertake this challenge. We sample crowd workers from 50 countries, have them score the same social media comments for toxicity and then evaluate the differences in the scores., altogether 18.,125 ratings. We find that the interpretation score differences among countries are highly significant. However., the hate interpretations vary more by the individual raters than by countries. These findings suggest that hate scoring systems should consider user-level features when scoring and automating the processing of online hate.",
author = "Joni Salminen and Fabio Veronesi and Hind Almerekhi and Jung, {Soon Gvo} and Jansen, {Bernard James}",
year = "2018",
month = "11",
day = "30",
doi = "10.1109/SNAMS.2018.8554954",
language = "English (US)",
series = "2018 5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "88--94",
booktitle = "2018 5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018",
address = "United States",

}

Salminen, J, Veronesi, F, Almerekhi, H, Jung, SG & Jansen, BJ 2018, Online Hate Interpretation Varies by Country, but More by Individual: A Statistical Analysis Using Crowdsourced Ratings. in 2018 5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018., 8554954, 2018 5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018, Institute of Electrical and Electronics Engineers Inc., pp. 88-94, 5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018, Valencia, Spain, 10/15/18. https://doi.org/10.1109/SNAMS.2018.8554954

Online Hate Interpretation Varies by Country, but More by Individual : A Statistical Analysis Using Crowdsourced Ratings. / Salminen, Joni; Veronesi, Fabio; Almerekhi, Hind; Jung, Soon Gvo; Jansen, Bernard James.

2018 5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 88-94 8554954 (2018 5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Online Hate Interpretation Varies by Country, but More by Individual

T2 - A Statistical Analysis Using Crowdsourced Ratings

AU - Salminen, Joni

AU - Veronesi, Fabio

AU - Almerekhi, Hind

AU - Jung, Soon Gvo

AU - Jansen, Bernard James

PY - 2018/11/30

Y1 - 2018/11/30

N2 - Hate is prevalent in online social media. This has resulted in a considerable amount of research in detecting and scoring it. Most computational efforts involve machine learning with crowdsourced ratings as training data. A prominent example of this is the Perspective API., a tool by Google to score toxicity of online comments. However., a major issue in the existing approaches is the lack of consideration for the subjective nature of online hate. While there is research that shows the intensity of hate varies and the hate depends on the context., there is no research that systematically investigates how hate interpretation varies by country or individual. In this exploratory research, we undertake this challenge. We sample crowd workers from 50 countries, have them score the same social media comments for toxicity and then evaluate the differences in the scores., altogether 18.,125 ratings. We find that the interpretation score differences among countries are highly significant. However., the hate interpretations vary more by the individual raters than by countries. These findings suggest that hate scoring systems should consider user-level features when scoring and automating the processing of online hate.

AB - Hate is prevalent in online social media. This has resulted in a considerable amount of research in detecting and scoring it. Most computational efforts involve machine learning with crowdsourced ratings as training data. A prominent example of this is the Perspective API., a tool by Google to score toxicity of online comments. However., a major issue in the existing approaches is the lack of consideration for the subjective nature of online hate. While there is research that shows the intensity of hate varies and the hate depends on the context., there is no research that systematically investigates how hate interpretation varies by country or individual. In this exploratory research, we undertake this challenge. We sample crowd workers from 50 countries, have them score the same social media comments for toxicity and then evaluate the differences in the scores., altogether 18.,125 ratings. We find that the interpretation score differences among countries are highly significant. However., the hate interpretations vary more by the individual raters than by countries. These findings suggest that hate scoring systems should consider user-level features when scoring and automating the processing of online hate.

UR - http://www.scopus.com/inward/record.url?scp=85060014671&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85060014671&partnerID=8YFLogxK

U2 - 10.1109/SNAMS.2018.8554954

DO - 10.1109/SNAMS.2018.8554954

M3 - Conference contribution

AN - SCOPUS:85060014671

T3 - 2018 5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018

SP - 88

EP - 94

BT - 2018 5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Salminen J, Veronesi F, Almerekhi H, Jung SG, Jansen BJ. Online Hate Interpretation Varies by Country, but More by Individual: A Statistical Analysis Using Crowdsourced Ratings. In 2018 5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 88-94. 8554954. (2018 5th International Conference on Social Networks Analysis, Management and Security, SNAMS 2018). https://doi.org/10.1109/SNAMS.2018.8554954