Abstract This study aimed to quantitatively determine the detection and evaluation capabilities of the ultrasonic procedure established by the American Welding Society in AWS D1.1 and D1.5 for the examination of welded steel bridges and buildings. Several reference steel blocks were manufactured employing ASTM A709 Grade 50W steel and including artificial reflectors with various shapes and dimensions, as follows: (1) with two types of planar artificial reflectors (Notches and Flat Bottom Holes), and (2) with one type of volumetric reflector (Side-Drilled Holes). The reflectors were manufactured in four sizes (0.5, 1, 3, and 5 mm), and were manually examined with a pulse-echo ultrasound equipment as follows: using the prescribed AWS procedure (Experiment 0), changing the crystal size (Experiment 1), and changing the frequency (Experiment 2). These modifications can induce changes in the detection and evaluation capabilities. Eight examination tests were performed for each type and size of artificial reflector, including variations in examination angle (45°, 60°, and 70°), as follows: two sound distances (first and second leg), two scanning levels, and two trials, for a total of 1224 measurements. Among the most important results, we found that the examination procedure prescribed by the AWS has good detection capability, which is not altered by changes in transducer frequency or crystal size. Nevertheless, this procedure has a poor ability to evaluate reflectors regardless of their size or type. In sum, with this methodology, it is almost impossible to determine, in a reliable way, the “severity” of natural discontinuities in welds or artificial reflectors.