We revisit an old idea that gravity can be unified with Yang-Mills theory by enlarging the gauge group of gravity formulated as gauge theory. Our starting point is an action that describes a generally covariant gauge theory for a group $G$. The Minkowski background breaks the gauge group by selecting in it a preferred gravitational SU(2) subgroup. We expand the action around this background and find the spectrum of linearized theory to consist of the usual gravitons plus Yang-Mills fields charged under the centralizer of the SU(2) in $G$. In addition, there is a set of Higgs fields that are charged both under the gravitational and Yang-Mills subgroups. These fields are generically massive and interact with both gravity and the Yang-Mills sector in the standard way. The arising interaction of the Yang-Mills sector with gravity is also standard. Parameters such as the Yang-Mills coupling constant and Higgs mass arise from the potential function defining the theory. Both are realistic in the sense explained in the paper.