Abstract Background An initiative aimed at improving nutrition intervention programs by providing food consumption data for better decision-making and promoting local foods has led to the development of an application-based food assessment tool called the Agrobiodiversity and Diet Diagnosis for Interventions Toolkit (ADD-IT). A new food frequency questionnaire (FFQ) was developed based on our previous study, which included 49 dishes and food items and asked about their frequency of consumption and portion size with photos. This study aimed to evaluate the validity of the FFQ. Methods We compared the estimated intake of energy, macro- and micronutrients, and food groups collected with the ADD-IT-based FFQ for one month with the estimated intake by means of nonconsecutive two-day 24-hour dietary recalls (24hDRs) of food consumption among women aged 19–70 years (mean = 33.7) in two rural areas—Kitui and Vihiga—in Kenya (N = 179). Spearman's rank correlation coefficients, cross-classification agreement, and Bland-Altman plots were used to assess the validity of the FFQ against the 24hDRs. Food consumption patterns for different communities were characterized by the FFQ and 24hDRs using the Mann-Whitney U test to ensure consistency in the results. Results The correlation coefficients ranged from low (-0.062) to moderate (0.479) between the FFQ and 24hDR. On median, approximately 71% of participants were correctly classified into the same or adjacent quartile, while 9% were misclassified by estimated nutrient intake from the FFQ. Bland-Altman plots ensured that there was no significant systematic bias for energy and macronutrients. The FFQ identified each community’s characteristic food consumption patterns as almost the same as those of the 24hDRs. Conclusions The newly developed app-based FFQ can be used to rank or classify individuals within each population according to their dietary intake and estimate the characteristics of their food intake patterns. However, selective reporting errors exist for specific foods. Several suggestions have been made to reduce such errors and improve the validation.