Automatic grading of programming assignments: An approach based on formal semantics

Xiao Liu, Shuai Wang, Pei Wang, Dinghao Wu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Programming assignment grading can be time-consuming and error-prone if done manually. Existing tools generate feedback with failing test cases. However, this method is inefficient and the results are incomplete. In this paper, we present AutoGrader, a tool that automatically determines the correctness of programming assignments and provides counterexamples given a single reference implementation of the problem. Instead of counting the passed tests, our tool searches for semantically different execution paths between a student's submission and the reference implementation. If such a difference is found, the submission is deemed incorrect; otherwise, it is judged to be a correct solution. We use weakest preconditions and symbolic execution to capture the semantics of execution paths and detect potential path differences. AutoGrader is the first automated grading tool that relies on program semantics and generates feedback with counterexamples based on path deviations. It also reduces human efforts in writing test cases and makes the grading more complete. We implement AutoGrader and test its effectiveness and performance with real-world programming problems and student submissions collected from an online programming site. Our experiment reveals that there are no false negatives using our proposed method and we detected 11 errors of online platform judges.

Original languageEnglish (US)
Title of host publicationProceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering
Subtitle of host publicationSoftware Engineering Education and Training, ICSE-SEET 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages126-137
Number of pages12
ISBN (Electronic)9781728110004
DOIs
StatePublished - May 1 2019
Event41st IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET 2019 - Montreal, Canada
Duration: May 25 2019May 31 2019

Publication series

NameProceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET 2019

Conference

Conference41st IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET 2019
CountryCanada
CityMontreal
Period5/25/195/31/19

Fingerprint

grading
Computer programming
programming
Semantics
semantics
Students
Feedback
student
experiment
performance
Experiments

All Science Journal Classification (ASJC) codes

  • Software
  • Education

Cite this

Liu, X., Wang, S., Wang, P., & Wu, D. (2019). Automatic grading of programming assignments: An approach based on formal semantics. In Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET 2019 (pp. 126-137). [8802114] (Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICSE-SEET.2019.00022
Liu, Xiao ; Wang, Shuai ; Wang, Pei ; Wu, Dinghao. / Automatic grading of programming assignments : An approach based on formal semantics. Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET 2019. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 126-137 (Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET 2019).
@inproceedings{3f61bbb8fac84a7995f1d96faccfc58f,
title = "Automatic grading of programming assignments: An approach based on formal semantics",
abstract = "Programming assignment grading can be time-consuming and error-prone if done manually. Existing tools generate feedback with failing test cases. However, this method is inefficient and the results are incomplete. In this paper, we present AutoGrader, a tool that automatically determines the correctness of programming assignments and provides counterexamples given a single reference implementation of the problem. Instead of counting the passed tests, our tool searches for semantically different execution paths between a student's submission and the reference implementation. If such a difference is found, the submission is deemed incorrect; otherwise, it is judged to be a correct solution. We use weakest preconditions and symbolic execution to capture the semantics of execution paths and detect potential path differences. AutoGrader is the first automated grading tool that relies on program semantics and generates feedback with counterexamples based on path deviations. It also reduces human efforts in writing test cases and makes the grading more complete. We implement AutoGrader and test its effectiveness and performance with real-world programming problems and student submissions collected from an online programming site. Our experiment reveals that there are no false negatives using our proposed method and we detected 11 errors of online platform judges.",
author = "Xiao Liu and Shuai Wang and Pei Wang and Dinghao Wu",
year = "2019",
month = "5",
day = "1",
doi = "10.1109/ICSE-SEET.2019.00022",
language = "English (US)",
series = "Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET 2019",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "126--137",
booktitle = "Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering",
address = "United States",

}

Liu, X, Wang, S, Wang, P & Wu, D 2019, Automatic grading of programming assignments: An approach based on formal semantics. in Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET 2019., 8802114, Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET 2019, Institute of Electrical and Electronics Engineers Inc., pp. 126-137, 41st IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET 2019, Montreal, Canada, 5/25/19. https://doi.org/10.1109/ICSE-SEET.2019.00022

Automatic grading of programming assignments : An approach based on formal semantics. / Liu, Xiao; Wang, Shuai; Wang, Pei; Wu, Dinghao.

Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 126-137 8802114 (Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Automatic grading of programming assignments

T2 - An approach based on formal semantics

AU - Liu, Xiao

AU - Wang, Shuai

AU - Wang, Pei

AU - Wu, Dinghao

PY - 2019/5/1

Y1 - 2019/5/1

N2 - Programming assignment grading can be time-consuming and error-prone if done manually. Existing tools generate feedback with failing test cases. However, this method is inefficient and the results are incomplete. In this paper, we present AutoGrader, a tool that automatically determines the correctness of programming assignments and provides counterexamples given a single reference implementation of the problem. Instead of counting the passed tests, our tool searches for semantically different execution paths between a student's submission and the reference implementation. If such a difference is found, the submission is deemed incorrect; otherwise, it is judged to be a correct solution. We use weakest preconditions and symbolic execution to capture the semantics of execution paths and detect potential path differences. AutoGrader is the first automated grading tool that relies on program semantics and generates feedback with counterexamples based on path deviations. It also reduces human efforts in writing test cases and makes the grading more complete. We implement AutoGrader and test its effectiveness and performance with real-world programming problems and student submissions collected from an online programming site. Our experiment reveals that there are no false negatives using our proposed method and we detected 11 errors of online platform judges.

AB - Programming assignment grading can be time-consuming and error-prone if done manually. Existing tools generate feedback with failing test cases. However, this method is inefficient and the results are incomplete. In this paper, we present AutoGrader, a tool that automatically determines the correctness of programming assignments and provides counterexamples given a single reference implementation of the problem. Instead of counting the passed tests, our tool searches for semantically different execution paths between a student's submission and the reference implementation. If such a difference is found, the submission is deemed incorrect; otherwise, it is judged to be a correct solution. We use weakest preconditions and symbolic execution to capture the semantics of execution paths and detect potential path differences. AutoGrader is the first automated grading tool that relies on program semantics and generates feedback with counterexamples based on path deviations. It also reduces human efforts in writing test cases and makes the grading more complete. We implement AutoGrader and test its effectiveness and performance with real-world programming problems and student submissions collected from an online programming site. Our experiment reveals that there are no false negatives using our proposed method and we detected 11 errors of online platform judges.

UR - http://www.scopus.com/inward/record.url?scp=85072117626&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85072117626&partnerID=8YFLogxK

U2 - 10.1109/ICSE-SEET.2019.00022

DO - 10.1109/ICSE-SEET.2019.00022

M3 - Conference contribution

AN - SCOPUS:85072117626

T3 - Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET 2019

SP - 126

EP - 137

BT - Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Liu X, Wang S, Wang P, Wu D. Automatic grading of programming assignments: An approach based on formal semantics. In Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 126-137. 8802114. (Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET 2019). https://doi.org/10.1109/ICSE-SEET.2019.00022