What Makes A Good Reference Manager? A Quantitative Analysis of Bibliography Management Applications

Many researchers and students use reference managers to collect, manage, and format references and citations. While prior work has assessed these tools qualitatively, it is still unclear how to quantitatively evaluate reference managers. This paper starts to quantify the user effort required to use reference managers. We first collected surveys from 69 graduate students to understand their experience with reference managers, and then conducted user studies with 12 participants. In our study, each participant was asked to perform a standardized task using four popular reference managers: Mendeley, Zotero, EndNote, and RefWorks. We used RUI, a keystroke and mouse-move logger, to record the participants' activities and approximate their physical and mental effort. We also used pre-and post-study surveys to collect users' feedback and self-reported task load (as expressed by the NASA TLX Index.) The results showed that different reference managers require different levels of effort, and users generally prefer the tools that involve less effort. We also found that although reference managers share similar features, differences in presentation and organization matter. We conclude this work by providing a set of guidelines for both users and developers of reference managers.