Ferroelectrics (FE) have been the materials of interest for non-volatile memories for many decades due to their hysteretic charge-voltage behavior. However, recently, the possibilities of integrating an FE in the gate stack of a transistor (forming a ferroelectric transistor or FEFET) have opened new avenues for computation and storage. The FEFETs not only enhance the design of non-volatile memories, but also lead to the unique possibilities of introducing non-volatility in close proximity with the compute elements. In this paper, we comparatively analyze several device and circuit aspects of FEFETs and FE capacitors from the perspective of designing non-volatile memory and logic. We discuss the effect of integrating an FE in a transistor structure on the remnant polarization and coercive voltage and show the importance of FE thickness optimization to design a non-volatile transistor. We also present circuit design possibilities with non-volatile FEFETs. First, the design of memories with separate read-write paths is discussed. We show that compared to FE capacitor based memories, FEFETs achieve enormous distinguishability and near read disturb free operation albeit with 2.5× higher cell area and 3.6× higher write energy at iso-write time. Second, we describe the opportunities that non-volatility combined with the three terminal architecture of FEFETs presents in the design of low power non-volatile flip-flops. We show that compared to FE capacitor based flip-flops, FEFET based design yields upto 50% lower energy and up to 40% lower delay for data back-up, along with 30% lower area.