We explore constraints on the scalar coupling in a family nonuniversal $U(1{)}^{\ensuremath{'}}$ extension of the standard model free from anomalies with a complex scalar dark matter particle. From unitarity and stability of the Higgs potential, we find the full set of bounds and order relations for the scalar coupling constants. Using recent data from the CERN-LHC collider, we study the signal strength of the diphoton Higgs decay, which imposes very stringent bounds to the scalar couplings and other scalar parameters, including parameters associated to the dark matter. Taking into account these constraints, the observable relic density of the Universe, and the limits from LUX collaboration for direct detection, we obtain allowed masses for the dark matter particle as low as 55 GeV. By assuming that the lightest scalar boson of the model corresponds to the observed Higgs boson, we evaluate deviations from the standard model of the trilineal Higgs self-coupling. The conditions from unitarity, stability and Higgs diphoton decay data allow trilineal deviations in the range $0\ensuremath{\le}\ensuremath{\delta}g\ensuremath{\lesssim}\ensuremath{-}72%$.