This paper presents a hierarchical approach to model tumor motion dynamics for image guided radiation therapy. Respiration induced tumor motion poses a significant challenge for using radiation therapy for tumors in the thorax and abdomen areas of the patients. The continuous motion of the tumor during radiation therapy can degrade the accuracy of radiation delivery and cause adverse effect on the surrounding healthy tissues. The proposed approach uses a two-stage modeling architecture-global modeling followed by local modeling to capture the dynamics of the tumor motion. The key idea of our proposed approach is based on an averaging method, which is able to merge arbitrary local models to a unbiased globally smooth model. Furthermore, the unscented Kalman filter is used to predict the tumor motion based on the identified nonlinear model and making use of respiratory motion observations in real-time. The proposed approach is tested by using numerical and experimental data. Our results show the proposed approach has a potential to achieve long-term tumor motion prediction with a sub-millimeter accuracy.