Application of MCAT questions as a testing tool and evaluation metric for knowledge graph–based reasoning systems

Karamarie Fecho, James Balhoff, Chris Bizon, William E. Byrd, Sui Hang, David Koslicki, Stefano E. Rensi, Patrick L. Schmitt, Mathias J. Wawer, Mark Williams, Stanley C. Ahalt

    Research output: Contribution to journalArticlepeer-review

    Abstract

    “Knowledge graphs” (KGs) have become a common approach for representing biomedical knowledge. In a KG, multiple biomedical data sets can be linked together as a graph representation, with nodes representing entities, such as “chemical substance” or “genes,” and edges representing predicates, such as “causes” or “treats.” Reasoning and inference algorithms can then be applied to the KG and used to generate new knowledge. We developed three KG-based question-answering systems as part of the Biomedical Data Translator program. These systems are typically tested and evaluated using traditional software engineering tools and approaches. In this study, we explored a team-based approach to test and evaluate the prototype “Translator Reasoners” through the application of Medical College Admission Test (MCAT) questions. Specifically, we describe three “hackathons,” in which the developers of each of the three systems worked together with a moderator to determine whether the applications could be used to solve MCAT questions. The results demonstrate progressive improvement in system performance, with 0% (0/5) correct answers during the first hackathon, 75% (3/4) correct during the second hackathon, and 100% (5/5) correct during the final hackathon. We discuss the technical and sociologic lessons learned and conclude that MCAT questions can be applied successfully in the context of moderated hackathons to test and evaluate prototype KG-based question-answering systems, identify gaps in current capabilities, and improve performance. Finally, we highlight several published clinical and translational science applications of the Translator Reasoners.

    Original languageEnglish (US)
    Pages (from-to)1719-1724
    Number of pages6
    JournalClinical and Translational Science
    Volume14
    Issue number5
    DOIs
    StatePublished - Sep 2021

    All Science Journal Classification (ASJC) codes

    • Neuroscience(all)
    • Biochemistry, Genetics and Molecular Biology(all)
    • Pharmacology, Toxicology and Pharmaceutics(all)

    Fingerprint

    Dive into the research topics of 'Application of MCAT questions as a testing tool and evaluation metric for knowledge graph–based reasoning systems'. Together they form a unique fingerprint.

    Cite this