Validity Concerns in Research Using Organic Data

Heng Xu, Nan Zhang, Le Zhou

Research output: Contribution to journalArticle

Abstract

With the advent of computing technologies, researchers across social science fields are using increasingly complex methods to collect, process, and analyze data in pursuit of scientific evidence. Given the complexity of research methods used, it is important to ensure that the research findings produced by a study are robust instead of being affected significantly by uncertainties associated with the design or implementation of the study. The field of metascience—the use of scientific methodology to study science itself—has examined various aspects of this robustness requirement for research that uses conventional designed studies (e.g., surveys, laboratory experiments) to collect data. Largely missing, however, are efforts to examine the robustness of empirical research using “organic data,” namely, data that are generated without any explicit research design elements and are continuously documented by digital devices (e.g., video captured by ubiquitous sensing devices; content and social interactions extracted from social networking sites, Twitter feeds, and click streams). Given the growing popularity of using organic data in management research, it is essential to understand issues concerning the usage and processing of organic data that may affect the robustness of research findings. This commentary first provides an overview of commonly present issues that threaten the validity of inferences drawn from empirical studies using organic data. This is followed by a discussion on some key considerations and suggestions for making organic data a robust and integral part of future research endeavors in management.

Original languageEnglish (US)
JournalJournal of Management
DOIs
StateAccepted/In press - Jan 1 2019

Fingerprint

Robustness
Social networking sites
Integral
Laboratory experiments
Social interaction
Management research
Empirical research
Uncertainty
Empirical study
Social sciences
Research design
Methodology
Inference
Science studies
Research methods
Twitter

All Science Journal Classification (ASJC) codes

  • Finance
  • Strategy and Management

Cite this

Xu, Heng ; Zhang, Nan ; Zhou, Le. / Validity Concerns in Research Using Organic Data. In: Journal of Management. 2019.
@article{1f65c459339441c3834f0a71736a286d,
title = "Validity Concerns in Research Using Organic Data",
abstract = "With the advent of computing technologies, researchers across social science fields are using increasingly complex methods to collect, process, and analyze data in pursuit of scientific evidence. Given the complexity of research methods used, it is important to ensure that the research findings produced by a study are robust instead of being affected significantly by uncertainties associated with the design or implementation of the study. The field of metascience—the use of scientific methodology to study science itself—has examined various aspects of this robustness requirement for research that uses conventional designed studies (e.g., surveys, laboratory experiments) to collect data. Largely missing, however, are efforts to examine the robustness of empirical research using “organic data,” namely, data that are generated without any explicit research design elements and are continuously documented by digital devices (e.g., video captured by ubiquitous sensing devices; content and social interactions extracted from social networking sites, Twitter feeds, and click streams). Given the growing popularity of using organic data in management research, it is essential to understand issues concerning the usage and processing of organic data that may affect the robustness of research findings. This commentary first provides an overview of commonly present issues that threaten the validity of inferences drawn from empirical studies using organic data. This is followed by a discussion on some key considerations and suggestions for making organic data a robust and integral part of future research endeavors in management.",
author = "Heng Xu and Nan Zhang and Le Zhou",
year = "2019",
month = "1",
day = "1",
doi = "10.1177/0149206319862027",
language = "English (US)",
journal = "Journal of Management",
issn = "0149-2063",
publisher = "SAGE Publications Inc.",

}

Validity Concerns in Research Using Organic Data. / Xu, Heng; Zhang, Nan; Zhou, Le.

In: Journal of Management, 01.01.2019.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Validity Concerns in Research Using Organic Data

AU - Xu, Heng

AU - Zhang, Nan

AU - Zhou, Le

PY - 2019/1/1

Y1 - 2019/1/1

N2 - With the advent of computing technologies, researchers across social science fields are using increasingly complex methods to collect, process, and analyze data in pursuit of scientific evidence. Given the complexity of research methods used, it is important to ensure that the research findings produced by a study are robust instead of being affected significantly by uncertainties associated with the design or implementation of the study. The field of metascience—the use of scientific methodology to study science itself—has examined various aspects of this robustness requirement for research that uses conventional designed studies (e.g., surveys, laboratory experiments) to collect data. Largely missing, however, are efforts to examine the robustness of empirical research using “organic data,” namely, data that are generated without any explicit research design elements and are continuously documented by digital devices (e.g., video captured by ubiquitous sensing devices; content and social interactions extracted from social networking sites, Twitter feeds, and click streams). Given the growing popularity of using organic data in management research, it is essential to understand issues concerning the usage and processing of organic data that may affect the robustness of research findings. This commentary first provides an overview of commonly present issues that threaten the validity of inferences drawn from empirical studies using organic data. This is followed by a discussion on some key considerations and suggestions for making organic data a robust and integral part of future research endeavors in management.

AB - With the advent of computing technologies, researchers across social science fields are using increasingly complex methods to collect, process, and analyze data in pursuit of scientific evidence. Given the complexity of research methods used, it is important to ensure that the research findings produced by a study are robust instead of being affected significantly by uncertainties associated with the design or implementation of the study. The field of metascience—the use of scientific methodology to study science itself—has examined various aspects of this robustness requirement for research that uses conventional designed studies (e.g., surveys, laboratory experiments) to collect data. Largely missing, however, are efforts to examine the robustness of empirical research using “organic data,” namely, data that are generated without any explicit research design elements and are continuously documented by digital devices (e.g., video captured by ubiquitous sensing devices; content and social interactions extracted from social networking sites, Twitter feeds, and click streams). Given the growing popularity of using organic data in management research, it is essential to understand issues concerning the usage and processing of organic data that may affect the robustness of research findings. This commentary first provides an overview of commonly present issues that threaten the validity of inferences drawn from empirical studies using organic data. This is followed by a discussion on some key considerations and suggestions for making organic data a robust and integral part of future research endeavors in management.

UR - http://www.scopus.com/inward/record.url?scp=85068888854&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85068888854&partnerID=8YFLogxK

U2 - 10.1177/0149206319862027

DO - 10.1177/0149206319862027

M3 - Article

AN - SCOPUS:85068888854

JO - Journal of Management

JF - Journal of Management

SN - 0149-2063

ER -