Background: The Household Dietary Diversity Score (HDDS) was developed to measure household food access, one of the levels of food security. Previous research has shown dietary diversity is related to food security. However, the validity of the HDDS in the form developed by the Food and Nutrition Technical Assistance (FANTA) project—12 food groups, 24-hour recall—and most frequently used by development organizations and nongovernmental organizations has never been verified. Objective: To verify the construct validity of the HDDS. Methods: A Rasch model was used to test the extent to which the HDDS meets the criteria required for interval scale measurement, using data from 1015 households in Colombia and Ecuador. Results: Different dietary patterns between Colombia and Ecuador and 2 cultural groups within Ecuador required data to be split into 3 subgroups. For each subgroup, the food groups meeting the criteria and their difficulty ranking were different. Refined indices, containing only those food groups meeting the criteria, contained 7 items in Colombia, 10 for Kichwa households in Ecuador, and 9 for migrant households. Conclusion: The indicator in its current form does not meet all criteria. Even when analyzing culturally homogenous subgroups within a small region, the components of the indicator do not form a reliable way of measuring household-level food access.