This paper describes a vision-based multiple model adaptive estimation using UAVs that enables the tracking of a mobile target that changes the system model depending on unknown factors. In our system the machine-learning-based target identification method uses Haar-like classifiers that detects the target position. The system uses multiple extended Kalman filters for each system model and estimates the states of the target through the observed positions. We estimate the probability that each system is true and use the max-probability method to determine the current model. The position controller of the UAVs uses the vision system not only to determine a desired waypoint but also to switch the control law for another model. Implementation of this system is validated through an image-in-the-loop simulation. We also explore an vision-based solution for Mission 7 of the international aerial robotics competition.