Automated content analysis: A case study of computer science student summaries

Yanjun Gao, Patricia M. Davies, Rebecca J. Passonneau

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

Technology is transforming Higher Education learning and teaching. This paper reports on a project to examine how and why automated content analysis could be used to assess précis writing by university students. We examine the case of one hundred and twenty-two summaries written by computer science freshmen. The texts, which had been hand scored using a teacher-designed rubric, were autoscored using the Natural Language Processing software, PyrEval. Pearsons correlation coefficient and Spearman rank correlation were used to analyze the relationship between the teacher score and the PyrEval score for each summary. Three content models automatically constructed by PyrEval from different sets of human reference summaries led to consistent correlations, showing that the approach is reliable. Also observed was that, in cases where the focus of student assessment centers on formative feedback, categorizing the PyrEval scores by examining the average and standard deviations could lead to novel interpretations of their relationships. It is suggested that this project has implications for the ways in which automated content analysis could be used to help university students improve their summarization skills.

Original languageEnglish (US)
Title of host publicationProceedings of the 13th Workshop on Innovative Use of NLP for Building Educational Applications, BEA 2018 at the 2018 Conference of the North American Chapter of the Association for Computational Linguistics
Subtitle of host publicationHuman Language Technologies, NAACL-HTL 2018
EditorsJoel Tetreault, Jill Burstein, Ekaterina Kochmar, Claudia Leacock, Helen Yannakoudakis
PublisherAssociation for Computational Linguistics (ACL)
Pages264-272
Number of pages9
ISBN (Electronic)9781948087117
StatePublished - 2018
Event13th Workshop on Innovative Use of NLP for Building Educational Applications, BEA 2018 at the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HTL 2018 - New Orleans, United States
Duration: Jun 5 2018 → …

Publication series

NameProceedings of the 13th Workshop on Innovative Use of NLP for Building Educational Applications, BEA 2018 at the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HTL 2018

Conference

Conference13th Workshop on Innovative Use of NLP for Building Educational Applications, BEA 2018 at the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HTL 2018
Country/TerritoryUnited States
CityNew Orleans
Period6/5/18 → …

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Artificial Intelligence
  • Language and Linguistics
  • Linguistics and Language

Cite this