A search is presented for anomalous quartic gauge boson couplings in vector-boson scattering. The data for the analysis correspond to 20.2 fb−1 of √s=8 TeV pp collisions and were collected in 2012 by the ATLAS experiment at the Large Hadron Collider. The search looks for the production of WW or WZ boson pairs accompanied by a high-mass dijet system, with one W decaying leptonically and a W or Z decaying hadronically. The hadronically decaying W/Z is reconstructed as either two small-radius jets or one large-radius jet using jet substructure techniques. Constraints on the anomalous quartic gauge boson coupling parameters α4 and α5 are set by fitting the transverse mass of the diboson system, and the resulting 95% confidence intervals are −0.024<α4<0.030 and −0.028<α5<0.033.Received 19 September 2016DOI:https://doi.org/10.1103/PhysRevD.95.032001Published by the American Physical Society under the terms of the Creative Commons Attribution 3.0 License. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.© 2017 CERN, for the ATLAS CollaborationPhysics Subject Headings (PhySH)Research AreasHypothetical particle physics modelsParticle interactionsParticles & Fields