Recent advances in digital photography and unmanned aerial vehicle (UAV) platforms allow people to easily and quickly record and acquire visual data from a construction site. To semi-automatically or automatically retrieve the needed information and assess the ongoing operations at a project level from the acquired visual data, researchers have focused on leveraging computer vision (CV) approaches to process and interpret visual data. This research focused on developing a framework to summarize the vision-based methods that have been applied to construction site operations through a literature review. The reviewed literature was composed of 55 journal papers from 2011 to 2019. The developed framework consisted of two parts: construction use cases and CV domains. Each part contained five aspects, which were integrated from the reviewed papers. For each reviewed paper, a map between these two parts was generated. A consistent definition was created for each term in the framework. The developed framework bridges the knowledge gap between construction and CV fields. It also highlights the frequency of applied computer vision-based methods in different construction use cases. Two additional use cases for future research were also identified.