Control of the motion of cavity solitons is one the central problems in nonlinear optical pattern formation. We report on the impact of the phase of the time-delayed optical feedback and carrier lifetime on the self-mobility of localized structures of light in broad-area semiconductor cavities. We show both analytically and numerically that the feedback phase strongly affects the drift instability threshold as well as the velocity of cavity soliton motion above this threshold. In addition we demonstrate that the noninstantaneous carrier response of the semiconductor medium is responsible for the increase of the critical feedback rate corresponding to the drift instability. © 2013 American Physical Society.