Small size microcavity pillars with elliptical cross section and high quality factors Q are reported and are shown to provide nearly 100% linearly polarized single photon sources. It is shown that the polarization of the emission of quantum dots embedded within the pillars can be controlled by using the coupling of the dot emission with the photonic modes. A notable dependence of the Q value is found on the polarization of the mode even though calculations of the mode profiles show that the electric field distribution is very similar.