Weather forecasting is a problem where an enormous amount of data must be processed. Severe storms cause a significant amount of damages and loss every year in part due to the insufficiency of the current techniques in producing reliable forecasts. We propose an algorithm that analyzes satellite images from the vast historical archives to predict severe storms. Conventional weather forecasting involves solving numerical models based on sensory data. It has been challenging for computers to make forecasts based on the visual patterns from satellite images. In our system we extract and summarize important visual storm evidence from satellite image sequences in a way similar to how meteorologists interpret these images. Particularly, the algorithm extracts and fits local cloud motions from image sequences to model the storm-related cloud patches. Image data of an entire year are adopted to train the model. The historical storm reports since the year 2000 are used as the ground-truth and statistical priors in the modeling process. Experiments demonstrate the usefulness and potential of the algorithm for producing improved storm forecasts.