Excessive amount of premature fruit drops can reach up to 25% of the entire production of the year. However, it is difficult to tell exact reasons of the premature drop due to the lack of a scientific diagnosis system. In this research, a machine vision system was developed to find where excessive fruit drop occurs in a citrus grove and to decide fruit dropping period by estimating that citrus was recently dropped or decayed due to being dropped. A hardware system was developed to facilitate automatic image acquisition in a citrus grove and reduce significant change in illumination in outdoor environment. A machine vision algorithm included thresholding and K-mean clustering to remove background and classification using a Random forest classifier. The result of this research shows that the correct identification was high (91.9%) for recently dropped citrus and relatively low (61.9%) for decayed fruit. Also, in-field spatial variability of the estimated amount of the dropped fruit by the algorithm was visualized on a map. This information can help growers to find potential causes of the fruit drop by correlating with other factors such as diseases, nutrition and soil types.