A search for neutral Higgs bosons decaying to tau pairs at a center-of-mass energy of 7 TeV is performed using a dataset corresponding to an integrated luminosity of 4.6 inverse femtobarns recorded by the CMS experiment at the LHC. The search is sensitive to both the standard model Higgs boson and to the neutral Higgs bosons predicted by the minimal supersymmetric extension of the standard model (MSSM). No excess of events is observed in the tau-pair invariant-mass spectrum. For a standard model Higgs boson in the mass range of 110-145 GeV upper limits at 95% confidence level (CL) on the production cross section are determined. We exclude a Higgs boson with m(H) = 115 GeV with a production cross section 3.2 times of that predicted by the standard model. In the MSSM, upper limits on the neutral Higgs boson production cross section times branching fraction to tau pairs, as a function of the pseudoscalar Higgs boson mass, m(A), sets stringent new bounds in the parameter space, excluding at 95% CL values of tan(beta) as low as 7.1 at m(A) = 160 GeV in the m[h](max) benchmark scenario.