We study inelastic processes in the hybrid quantum system constituted by a carbon nanotube (CNT) in contact with an ultracold quantum gas, such as a cloud of thermal atoms or a Bose-Einstein condensate (BEC).We present a parameter-free ab-initio approach for the loss rate based on the underlying scattering process, considering the two-dimensional character of the system as well as the exact Casimir-Polder potential.The predicted loss rates are in perfect agreement with recent experimental results, obtained both for a thermal cloud of rubidium atoms and for a BEC.For the trap loss of a thermal cloud, we find that retardation effects become important and contribute significantly, which emphasises the crucial role of the exact interaction potential.