Electromagnetic observations have been used over the past decades to understand the nature of black holes and the material around them. Our ability to learn about the fundamental physics relies on our understanding of two key ingredients in the modeling of these electromagnetic observations: the gravity theory that describes the black hole, and the astrophysics that produces the observed radiation. In this work we study our current ability to constrain and detect deviations from general relativity using the accretion disk spectrum of stellar-mass black holes in binary systems. Our analysis combines relativistic ray-tracing and Markov-Chain Monte-Carlo sampling techniques to determine how well such tests of General Relativity can be carried out in practice. We show that even when a very simple astrophysical model for the accretion disk is assumed a priori, the uncertainties and covariances between the parameters of the model and the parameters that control the deformation from general relativity make any test of general relativity very challenging with accretion disk spectrum observations. We also discuss the implications of assuming that general relativity is correct a priori on the estimation of parameters of the astrophysical model when the data is not described by Einstein's theory, which can lead to a fundamental systematic bias.