Compared with previous graphene-based optical modulators that can be tuned electrically with the Fermi level, all-optical modulators have attracted much attention due to their ultrafast and broadband response (bandwidth ≈ 200 GHz). To take advantage of hybrid plasmonic waveguides (HPWs) and the nonlinear response of graphene, we proposed all-optical absorption and phase modulators at the wavelength of 1.55 μm and E_F = 0.1 eV. The modulators consist of two different geometries, graphene on an HPW and graphene on a double-slot HPW. Compared with the previous graphene-based all-optical modulators, our design has a short-effective waveguide length and higher modulation depth. This was achieved through the incomparable integration of HPW optics and graphene photonics that can be used to design chip-scale, compact, and high-efficiency all-optical modulators.