This paper explores how edge serrations on a truncated ground plane affect the performance of a microstrip antenna. The investigation includes both simulations and measurements of a range of serration dimensions. Improvements in performance through ground plane edge serration include lower cross-polarization level, increased beam width, and slight impedance tuning compared to a microstrip antenna on a simple finite ground plane.
Huff, G. H., & Bernhard, J. T. (2002). Improvements in the performance of microstrip antennas on finite ground planes through ground plane edge serrations. IEEE Microwave and Wireless Components Letters, 12(8), 308-310. https://doi.org/10.1109/LMWC.2002.802032