Atrial Fibrillation is a common cardiac arrhythmia, which is characterized by an abnormal heartbeat rhythm that can be life-threatening. Recently, researchers have proposed several Convolutional Neural Networks (CNNs) to detect Atrial Fibrillation. CNNs have high requirements on computing and memory resources, which usually demand the use of High Performance Computing (eg, GPUs). This high energy demand is a challenge for portable devices. Therefore, efficient hardware implementations are required. We propose a computational architecture for the inference of a Quantized Convolutional Neural Network (Q-CNN) that allows the detection of the Atrial Fibrillation (AF). The architecture exploits data-level parallelism by incorporating SIMD-based vector units, which is optimized in terms of computation and storage and also optimized to perform both the convolutional and fully connected layers. The computational architecture was implemented and tested in a Xilinx Artix-7 FPGA. We present the experimental results regarding the quantization process in a different number of bits, hardware resources, and precision. The results show an accuracy of 94% accuracy for 22-bits. This work aims to be the basis for the future implementation of a portable, low-cost, and high-reliability device for the diagnosis of Atrial Fibrillation.