We consider optimal control synthesis problems for control-affine systems under safety constraints. We are interested in the task of optimally driving a system from an initial set to a desired state and avoiding some predefined unsafe sets at the same time. Building on a weaker notion of stability of dynamic systems relying on the Lyapunov density, we formulate the optimal control synthesis problem into a convex optimization problem in the space of densities. In this formulation, the safety constraints are naturally mapped to convex constraints on the densities. To improve scalability, the densities are parameterized by polynomials or rational functions. The Sum-of-Squares technique is then employed to efficiently solve the optimal control problems and verify the safety constraints. Numerical examples are presented to demonstrate the efficacy of the proposed framework.