Underwater object detection is an important computer vision task that has been widely used in marine life identification and tracking. However, there are a series of challenges such as low-contrast condition, occlusion condition, unbalanced light condition, and small dense objects in underwater object detection. Attention mechanism has been proven powerful in feature extraction. However, attention mechanisms usually divide image into fixed patches. This methods lead to the splitting up of continuous structures, which hinders the use of similar information in other areas to enhance image details. In fact, using graph structure is more flexible and effective for visual perception, because it can capture complex target relation and context information effectively. Thus, we apply graph attention mechanisms to irregular patches and propose an irregular-patch graph attention network (IGA-Net). First, the superpixel segmentation method is used to segment the image to reduce noise. Second, the global graph and local graph are constructed to obtain internal structures. Finally, to handle occlusion and small objects, a distinctive feature three-way handshake (F3H) module is proposed to fuse information from global and local graph. To demonstrate the effectiveness of the proposed method, we conduct comprehensive evaluations on five challenging underwater datasets UTDAC2020, RUOD, Brackish, TrashCan, and WPBB. Experimental results demonstrate that the proposed IGA-Net achieves superior performance on five challenging underwater datasets.