The packing of aggregate skeleton directly affects the capability of a mixture to transmit and distribute loads. It has significant impact on the performance hot mix asphalt (HMA) mixtures. This paper conducted an analysis of aggregate packing characteristics using a Discrete Element Modeling (DEM) simulation. A Washington typical HMA gradation is used as an example for the analysis. By using the PFC3D DEM simulation to relate the gradation parameter to the volumetric properties of the structure, this paper theoretically demonstrated the different roles of aggregate particles in an HMA mixture. A new criterion to differentiate "coarse" and "fine" aggregates is developed, which is consistent with the Bailey's method definition. Aggregate contacts and contact force are also studied in this paper to evaluate the capability of the aggregate structure to transmit stresses through aggregate skeleton, and thereby, to resist permanent deformation. It is suggested that the method described in this paper based on the analysis of aggregate packing and DEM simulation provide a promising direction to theoretically understand the characteristics of a packed aggregate structure, as well as to guide the aggregate gradation design.