The majority of current models utilized for predicting toxicity in prostate cancer radiotherapy are based on dose-volume histograms. One of their main drawbacks is the lack of spatial accuracy, since they consider the organs as a whole volume and thus ignore the heterogeneous intra-organ radio-sensitivity. In this paper, we propose a dose-image-based framework to reveal the relationships between local dose and toxicity. In this approach, the three-dimensional (3D) planned dose distributions across a population are non-rigidly registered into a common coordinate system and compared at a voxel level, therefore enabling the identification of 3D anatomical patterns, which may be responsible for toxicity, at least to some extent. Additionally, different metrics were employed in order to assess the quality of the dose mapping. The value of this approach was demonstrated by prospectively analyzing rectal bleeding (≥Grade 1 at 2 years) according to the CTCAE v3.0 classification in a series of 105 patients receiving 80 Gy to the prostate by intensity modulated radiation therapy (IMRT). Within the patients presenting bleeding, a significant dose excess (6 Gy on average, p < 0.01) was found in a region of the anterior rectal wall. This region, close to the prostate (1 cm), represented less than 10% of the rectum. This promising voxel-wise approach allowed subregions to be defined within the organ that may be involved in toxicity and, as such, must be considered during the inverse IMRT planning step.