The permeability of fractured sorbing media, such as shale and coal, is mainly controlled by effective stresses and sorption-induced strains. Although the influence of effective stresses on permeability has been extensively studied, how sorption-induced strains affect permeability evolution has not been fully understood. Sorption-induced strains can impact the permeability in opposite ways at different time scales. If the swelling occurs at matrix surfaces (local swelling), the swelling strain purely reduces fracture aperture and results in a permeability decline. However, when the whole rock is fully invaded by injected gas, the swelling of the whole rock (global swelling) increases fracture aperture and the bulk volume, enhancing the permeability. Most existing models only use fracture (pore) pressure to describe rock swelling, assuming that the rock is fully invaded and matrix-fracture pressure equilibrium is achieved. They cannot explain some experimental data because rocks may never be fully invaded during permeability measurement. Moreover, different pore types are not considered and local swelling can be heterogeneous due to complex matrix components. In this paper, the non-equilibrium effects are depicted by defining two continua (matrices and fractures) with distinct pressure values. The transition between local swelling and global swelling is quantified by the pressure difference between the two systems. The larger the pressure difference is, the heavier local swelling effects will be. And global swelling is only a function of fracture pressure. Different pore types are included in our permeability model. And the heterogeneous local swelling strain is characterized by a splitting strain function. This model is verified against laboratory data from common permeability measurement conditions. Under constant effective stress and constant confining stress conditions, the permeability changes at different times and becomes stable after a relatively long time. With the matrix-fracture pressure difference first increases to a maximum value and then decreases to zero, local swelling effects change from zero to a peak value and finally drop to zero. By combining permeability curves at different injection pressure levels, 3-D permeability surfaces are obtained. The impacts of rock properties, heterogeneous local swelling, and multiple pore types on permeability evolution are analyzed. Adsorption and mechanical properties control specific regions of permeability curves. Effects of heterogeneous local swelling are determined by the adsorption capacity of the dominant matrix component. The existence of multiple pore types makes the permeability curve deviate from those of single-pore-type cases and affects a wider range of permeability curves compared with heterogeneous local swelling.