In this paper, we study the following rigid substructure pattern reconstruction problem: given a set of n input structures (i.e. point-sets), partition each structure into k rigid sub-structures so that the nk rigid substructures can be grouped into k clusters with each of them containing exact one rigid substructure from every input structure and the total clustering cost is minimized, where the clustering cost of a cluster is the total distance between a pattern reconstructed for this cluster and every member rigid substructure. Different from most of the existing models for pattern reconstruction (where each input point-set is often treated as a single structure), our model views each input point-set as a collection of k rigid substructures, and aims to extract similar rigid substructures from each input point-set to form k rigid clusters. The problem is motivated by an interesting biological application for determining the topological structure of chromosomes inside the cell nucleus. We propose a highly effective and practical solution based on a number of new insights to pattern reconstruction, clustering, and motion detection. We validate our method on synthetic, biological and motion tracking datasets. Experimental results suggest that our approach yields a near optimal solution.