A rapid reduction in sediment porosity from 60 to 70 % at seafloor to less than 10 % at several kilometers depth can play an important role in deformation and seismicity in the shallow portion of subduction zones. We conducted deformation experiments on rocks from an ancient accretionary complex, the Shimanto Belt, across the Nobeoka Thrust to understand the deformation behaviors of rocks along plate boundary faults at seismogenic depth. Our experimental results for phyllites in the hanging wall and shale-tuff mélanges in the footwall of the Nobeoka Thrust indicate that the Shimanto Belt rocks fail brittlely accompanied by a stress drop at effective pressures < 80 MPa, whereas they exhibit strain hardening at higher effective pressures. The transition from brittle to ductile behavior in the shale–tuff mélanges lies on the same trend in effective stress–porosity space as that for clay-rich and tuffaceous sediments subducting into the modern Nankai subduction zone. Both the absolute yield strength and the effective pressure at the brittle–ductile transition for the phyllosilicate-rich materials are much lower than for sandstones. These results suggest that as the clay-rich or tuffaceous sediments subduct and their porosities are reduced, their deformation behavior gradually transitions from ductile to brittle and their yield strength increases. Our results also suggest that samples of the ancient Shimanto accretionary prism can serve as an analog for underthrust rocks at seismogenic depth in the modern Nankai Trough.