It is often assumed that the exchange interaction between two magnetic ions in a semiconductor host depends only on the distance and orientation of the magnetic ions. Using first-principles electronic structure calculations of Mn impurities in GaAs, we show that the exchange interaction between two magnetic ions depends also on the concentration and spatial arrangement of the other, ``spectator'' magnetic ions. Thus, such systems cannot be described by a Heisenberg Hamiltonian with fixed exchange interactions. Specifically, we find that at fixed Mn concentration, association (``clustering'') of Mn impurities leads to a decrease of the Curie temperature, while dissociation (``declustering'') leads to higher Curie temperatures. We conclude that clustering is the major impediment to achieve high Curie temperatures in Mn-doped GaAs.