Infant brain imaging is highly challenging but necessary for diagnosing various prevalent disorders including vascular malformations, encephalitis, and abusive head trauma. Conventional brain imaging technologies such as MRI, CT, and PET are not suitable for repeated use on neonates due to the use of ionizing radiation (CT and PET), need for patient transport, uncomfortable environment, high cost, and bulky equipment. A wearable photoacoustic imaging (PAI) hat can be an ideal candidate for this application. However, its practical realization suffers from many system design problems such as complex assembly, unviability of full-hat rotation around the neonatal head, ultrasound coupling, and requirements of <3,000 ultrasound data acquisition channels to cover the whole brain. Here, we present a modular photoacoustic imaging (PAI) hat solution that uses an innovative modular design approach, making it realizable by assembling individual working units while minimizing the challenges of back-end electronics. The modular photoacoustic hat consists of multiple PAI disc modules of 2 inches in diameter that conform to the shape of the local head surface and assembled on a hat to cover the whole neonatal brain. Each PAI disc is integrated with optical fibers for light excitation of brain tissue. For photoacoustic detection, the discs are either densely packed with ultrasound elements to eliminate the need for rotation or can have fewer ultrasound elements (usually in trapezoidal shape) on the rotating disc to overcome large number of data acquisition channels. In this article, we have demonstrated the design, integration and initial results of the proposed wearable PAI-hat.