We report the design of a highly efficient, compact and ultra-broadband hybrid graphene-silicon modulator. The design comprises a single graphene layer deposited on a silicon waveguide, using a thin hafnium dioxide dielectric spacer. The light-graphene interaction was tuned, by geometric optimization, in order to provide polarization-insensitive modulation. The simulation results indicate high speed polarization-insensitive modulation, in amplitude or phase, within the entire optical communications range. The energy per bit (Ebit) was also calculated for both modulation mechanisms, indicating a way to optimize the energy consumption according to the desired application. We expect that the fabrication of the proposed device helps to pave the way to ultra-broadband optical interconnects that can cope with the growing demand for higher capacity and bandwidth in optical communications.