The hippocampus, a crucial structure in the brain, plays a significant role in the early diagnosis of brain disorders such as Alzheimer’s disease through its structural and volumetric changes. To address the medical challenge of accurately segmenting the hippocampus, we propose a lightweight hybrid segmentation network called a parallel cascaded feature reconstruction network (PCFR-Net). This network integrates the advantages of global self-attention and local convolution while utilizing fewer model parameters. Specifically, we introduce a feature reconstruction (FR) module and a multibranch asymmetric residual attention module aimed at accurate segmentation of hippocampus magnetic resonance imaging. The model combines the strengths of the transformer in capturing long-distance relationships and adapting to irregular shapes, as well as the FR block, which can reduce the redundancy in space and channels during feature extraction, and then reconstructs feature maps to enhance the representative feature learning. In addition, the multibranch residual attention module employs the asymmetric residual convolution block, enabling fine-grained feature extraction along the length, width, and depth directions at multiple scales. Remarkably, the proposed PCFR-Net achieves a Dice similarity coefficient (DSC) of 92.74% and an Intersection over Union (IoU) of 86.5% on the Medical Segmentation Decathlon, as well as a DSC of 93.86% and an IoU of 89.29% on the Alzheimer’s Disease Neuroimaging Initiative dataset.