Low-cost, instrument-free colorimetric tests were developed to detect SARS-CoV-2 using plasmonic biosensors with Au nanoparticles functionalized with polyclonal antibodies (f-AuNPs). Intense color changes were noted with the naked eye owing to plasmon coupling when f-AuNPs form clusters on the virus, with high sensitivity and a detection limit of 0.28 PFU mL-1 (PFU stands for plaque-forming units) in human saliva. Plasmon coupling was corroborated with computer simulations using the finite-difference time-domain (FDTD) method. The strategies based on preparing plasmonic biosensors with f-AuNPs are robust to permit SARS-CoV-2 detection via dynamic light scattering and UV-vis spectroscopy without interference from other viruses, such as influenza and dengue viruses. The diagnosis was made with a smartphone app after processing the images collected from the smartphone camera, measuring the concentration of SARS-CoV-2. Both image processing and machine learning algorithms were found to provide COVID-19 diagnosis with 100% accuracy for saliva samples. In subsidiary experiments, we observed that the biosensor could be used to detect the virus in river waters without pretreatment. With fast responses and requiring small sample amounts (only 20 μL), these colorimetric tests can be deployed in any location within the point-of-care diagnosis paradigm for epidemiological control.